2024-05-26T11:41:24.209 INFO:root:teuthology version: 0.0.1.dev291+ge7ae758 2024-05-26T11:41:24.209 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} --verbose --owner scheduled_leonidus@teuthology --archive /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315 --name leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi -- /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/orig.config.yaml 2024-05-26T11:41:24.262 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-26T11:41:24.348 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315 branch: wip-lusov-quiesce description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} email: null first_in_suite: false job_id: '7726315' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-lusov-quiesce ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-lusov-quiesce sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 owner: scheduled_leonidus@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2532 sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 sleep_before_teardown: 0 subset: 345/512 suite: fs suite_branch: wip-lusov-quiesce suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi156.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHPUrTLMrpH8LQVaQNItH/JpRKe4iFTkWmSFyUD+9SHDy+GfStPSGLe9Is3L9ZkEdSCmKycvjcceYarL03DON9M= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDT584TpMcRlKuumRLYzQ3VdFYWkecJyMHTsuvyA4gGhNUIzUmPyh3qtrHGkfDybHeer/NNTvCFaOI2eNZcQb5M= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIV/tEjw/jIsRH6d7qmlTAcpBZ4KXPLWCJo+wPEJ50qkPMcElPvjAsED0NmgyRcqpAJkry66JoqxrBth+vozwSg= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - quiescer: null - workunit: clients: all: - kernel_untar_build.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e7ae7580c417d6ab27fda98fecadca5d9f85d63e timestamp: 2024-05-26_10:47:39 tube: smithi user: leonidus verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-05-26T11:41:24.348 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/qa; will attempt to use it 2024-05-26T11:41:24.349 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/qa/tasks 2024-05-26T11:41:24.349 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-26T11:41:24.351 INFO:teuthology.task.internal:Checking packages... 2024-05-26T11:41:24.371 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209' 2024-05-26T11:41:24.371 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-26T11:41:24.371 INFO:teuthology.packaging:ref: None 2024-05-26T11:41:24.371 INFO:teuthology.packaging:tag: None 2024-05-26T11:41:24.371 INFO:teuthology.packaging:branch: wip-lusov-quiesce 2024-05-26T11:41:24.371 INFO:teuthology.packaging:sha1: 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:41:24.372 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-lusov-quiesce 2024-05-26T11:41:24.813 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3867.g9a4c5853 2024-05-26T11:41:24.815 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-26T11:41:24.821 INFO:teuthology.task.internal:no buildpackages task found 2024-05-26T11:41:24.822 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-26T11:41:24.826 INFO:teuthology.task.internal:Saving configuration 2024-05-26T11:41:24.845 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-26T11:41:24.852 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-26T11:41:24.880 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi156.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 11:36:16.388953', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHPUrTLMrpH8LQVaQNItH/JpRKe4iFTkWmSFyUD+9SHDy+GfStPSGLe9Is3L9ZkEdSCmKycvjcceYarL03DON9M='} 2024-05-26T11:41:24.904 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 11:36:16.390117', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDT584TpMcRlKuumRLYzQ3VdFYWkecJyMHTsuvyA4gGhNUIzUmPyh3qtrHGkfDybHeer/NNTvCFaOI2eNZcQb5M='} 2024-05-26T11:41:24.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-26 11:36:16.391064', 'locked_by': 'scheduled_leonidus@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIV/tEjw/jIsRH6d7qmlTAcpBZ4KXPLWCJo+wPEJ50qkPMcElPvjAsED0NmgyRcqpAJkry66JoqxrBth+vozwSg='} 2024-05-26T11:41:24.927 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-26T11:41:24.934 INFO:teuthology.task.internal:roles: ubuntu@smithi156.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-26T11:41:24.934 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-26T11:41:24.934 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-26T11:41:24.934 INFO:teuthology.run_tasks:Running task console_log... 2024-05-26T11:41:25.019 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5196ae3ac0>, signals=[15]) 2024-05-26T11:41:25.019 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-26T11:41:25.025 INFO:teuthology.task.internal:Opening connections... 2024-05-26T11:41:25.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi156.front.sepia.ceph.com 2024-05-26T11:41:25.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:41:25.101 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2024-05-26T11:41:25.102 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:41:25.174 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-05-26T11:41:25.175 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:41:25.250 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-26T11:41:25.257 DEBUG:teuthology.orchestra.run.smithi156:> uname -m 2024-05-26T11:41:25.275 INFO:teuthology.orchestra.run.smithi156.stdout:x86_64 2024-05-26T11:41:25.275 DEBUG:teuthology.orchestra.run.smithi156:> cat /etc/os-release 2024-05-26T11:41:25.333 INFO:teuthology.orchestra.run.smithi156.stdout:NAME="CentOS Stream" 2024-05-26T11:41:25.333 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION="9" 2024-05-26T11:41:25.333 INFO:teuthology.orchestra.run.smithi156.stdout:ID="centos" 2024-05-26T11:41:25.333 INFO:teuthology.orchestra.run.smithi156.stdout:ID_LIKE="rhel fedora" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:VERSION_ID="9" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:PLATFORM_ID="platform:el9" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:ANSI_COLOR="0;31" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:LOGO="fedora-logo-icon" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:HOME_URL="https://centos.org/" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-26T11:41:25.334 INFO:teuthology.orchestra.run.smithi156.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-26T11:41:25.335 INFO:teuthology.lock.ops:Updating smithi156.front.sepia.ceph.com on lock server 2024-05-26T11:41:25.360 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2024-05-26T11:41:25.377 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2024-05-26T11:41:25.377 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2024-05-26T11:41:25.434 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2024-05-26T11:41:25.434 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-26T11:41:25.435 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-26T11:41:25.436 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2024-05-26T11:41:25.451 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-05-26T11:41:25.469 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-05-26T11:41:25.469 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-05-26T11:41:25.526 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-26T11:41:25.527 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-05-26T11:41:25.527 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-26T11:41:25.527 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-26T11:41:25.527 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-26T11:41:25.527 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-05-26T11:41:25.551 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-26T11:41:25.561 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-26T11:41:25.616 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-26T11:41:25.617 DEBUG:teuthology.orchestra.run.smithi156:> test '!' -e /home/ubuntu/cephtest 2024-05-26T11:41:25.619 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2024-05-26T11:41:25.621 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-05-26T11:41:25.638 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-26T11:41:25.644 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-26T11:41:25.644 DEBUG:teuthology.orchestra.run.smithi156:> test -z $(ls -A /var/lib/ceph) 2024-05-26T11:41:25.676 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2024-05-26T11:41:25.678 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-05-26T11:41:25.807 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-26T11:41:25.885 INFO:teuthology.run_tasks:Running task kernel... 2024-05-26T11:41:25.902 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-05-26T11:41:25.902 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-26T11:41:25.902 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-26T11:41:25.903 DEBUG:teuthology.orchestra.run.smithi156:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-26T11:41:25.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:41:25.920 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-05-26T11:41:25.977 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-437.el9.x86_64 2024-05-26T11:41:25.977 INFO:teuthology.task.kernel:Running kernel on smithi156: 5.14.0-437.el9.x86_64 2024-05-26T11:41:25.978 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum install -y kernel 2024-05-26T11:41:29.605 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.1 MB 00:01 2024-05-26T11:41:31.315 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-05-26T11:41:37.240 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.2 MB 00:02 2024-05-26T11:41:39.099 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 17 kB 00:00 2024-05-26T11:41:40.067 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-05-26T11:41:45.309 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 21 kB/s | 1.7 kB 00:00 2024-05-26T11:41:47.190 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-26T11:41:47.190 INFO:teuthology.orchestra.run.smithi156.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-26T11:41:47.285 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout: Package Architecture Version Repository Size 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout: kernel x86_64 5.14.0-447.el9 baseos 576 k 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core x86_64 5.14.0-447.el9 baseos 16 M 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules x86_64 5.14.0-447.el9 baseos 34 M 2024-05-26T11:41:47.294 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core x86_64 5.14.0-447.el9 baseos 28 M 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:================================================================================ 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:Install 4 Packages 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 78 M 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:Installed size: 123 M 2024-05-26T11:41:47.295 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-05-26T11:41:47.510 INFO:teuthology.orchestra.run.smithi156.stdout:(1/4): kernel-5.14.0-447.el9.x86_64.rpm 3.1 MB/s | 576 kB 00:00 2024-05-26T11:41:48.144 INFO:teuthology.orchestra.run.smithi156.stdout:(2/4): kernel-core-5.14.0-447.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-05-26T11:41:48.777 INFO:teuthology.orchestra.run.smithi156.stdout:(3/4): kernel-modules-5.14.0-447.el9.x86_64.rpm 24 MB/s | 34 MB 00:01 2024-05-26T11:41:49.036 INFO:teuthology.orchestra.run.smithi156.stdout:(4/4): kernel-modules-core-5.14.0-447.el9.x86_6 18 MB/s | 28 MB 00:01 2024-05-26T11:41:49.036 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-05-26T11:41:49.036 INFO:teuthology.orchestra.run.smithi156.stdout:Total 45 MB/s | 78 MB 00:01 2024-05-26T11:41:49.723 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-05-26T11:41:50.059 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-05-26T11:41:50.059 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-05-26T11:41:50.987 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-05-26T11:41:50.987 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-05-26T11:41:53.386 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-05-26T11:41:54.034 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-core-5.14.0-447.el9.x86_64 1/4 2024-05-26T11:41:54.137 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-26T11:41:56.811 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-26T11:41:56.959 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-26T11:42:01.669 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-26T11:42:01.878 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : kernel-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:06.632 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:36.108 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-core-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:36.146 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-modules-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:36.429 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: kernel-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:36.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-5.14.0-447.el9.x86_64 1/4 2024-05-26T11:42:36.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-core-5.14.0-447.el9.x86_64 2/4 2024-05-26T11:42:36.429 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-5.14.0-447.el9.x86_64 3/4 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : kernel-modules-core-5.14.0-447.el9.x86_64 4/4 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-core-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: kernel-modules-core-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:42:37.238 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-05-26T11:42:37.576 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T11:42:37.662 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.663 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-26T11:42:37.663 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-447.el9.x86_64 2024-05-26T11:42:37.663 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-26T11:42:37.663 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-26T11:42:37.663 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-26T11:42:37.663 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi156.front.sepia.ceph.com, path=None, version=distro) 2024-05-26T11:42:37.663 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q kernel | sort -rV | head -n 1 2024-05-26T11:42:37.713 INFO:teuthology.orchestra.run.smithi156.stdout:kernel-5.14.0-447.el9.x86_64 2024-05-26T11:42:37.713 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-447.el9.x86_64 2024-05-26T11:42:37.713 DEBUG:teuthology.orchestra.run.smithi156:> sudo rpm -qi grub2-tools 2024-05-26T11:42:37.754 INFO:teuthology.orchestra.run.smithi156.stdout:Name : grub2-tools 2024-05-26T11:42:37.754 INFO:teuthology.orchestra.run.smithi156.stdout:Epoch : 1 2024-05-26T11:42:37.754 INFO:teuthology.orchestra.run.smithi156.stdout:Version : 2.06 2024-05-26T11:42:37.754 INFO:teuthology.orchestra.run.smithi156.stdout:Release : 68.el9 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Architecture: x86_64 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Group : Unspecified 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Size : 8264254 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:License : GPLv3+ 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Packager : builder@centos.org 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Vendor : CentOS 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Summary : Support tools for GRUB. 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:Description : 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:42:37.755 INFO:teuthology.orchestra.run.smithi156.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-26T11:42:37.756 INFO:teuthology.orchestra.run.smithi156.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-26T11:42:37.756 INFO:teuthology.orchestra.run.smithi156.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-26T11:42:37.756 INFO:teuthology.orchestra.run.smithi156.stdout:hardware devices. 2024-05-26T11:42:37.756 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:42:37.756 INFO:teuthology.orchestra.run.smithi156.stdout:This subpackage provides tools for support of all platforms. 2024-05-26T11:42:37.757 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-26T11:42:37.757 INFO:teuthology.task.kernel:Updating grub on smithi156 to boot 5.14.0-447.el9.x86_64 2024-05-26T11:42:37.757 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-26T11:42:38.385 INFO:teuthology.orchestra.run.smithi156.stderr:Generating grub configuration file ... 2024-05-26T11:42:39.774 INFO:teuthology.orchestra.run.smithi156.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-26T11:42:39.808 INFO:teuthology.orchestra.run.smithi156.stderr:done 2024-05-26T11:42:39.810 DEBUG:teuthology.orchestra.run.smithi156:> mktemp 2024-05-26T11:42:39.826 INFO:teuthology.orchestra.run.smithi156.stdout:/tmp/tmp.EMVdsNzuyF 2024-05-26T11:42:39.826 DEBUG:teuthology.orchestra.run.smithi156:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.EMVdsNzuyF 2024-05-26T11:42:39.899 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0666 /tmp/tmp.EMVdsNzuyF 2024-05-26T11:42:40.039 DEBUG:teuthology.orchestra.remote:smithi156:/tmp/tmp.EMVdsNzuyF is 6KB 2024-05-26T11:42:40.090 DEBUG:teuthology.orchestra.run.smithi156:> rm -fr /tmp/tmp.EMVdsNzuyF 2024-05-26T11:42:40.104 DEBUG:teuthology.orchestra.run.smithi156:> sudo /bin/ls /boot/loader/entries || true 2024-05-26T11:42:40.165 INFO:teuthology.orchestra.run.smithi156.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-26T11:42:40.166 INFO:teuthology.orchestra.run.smithi156.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-26T11:42:40.166 INFO:teuthology.orchestra.run.smithi156.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-26T11:42:40.166 INFO:teuthology.orchestra.run.smithi156.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64.conf 2024-05-26T11:42:40.167 DEBUG:teuthology.orchestra.run.smithi156:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-447.el9.x86_64 2024-05-26T11:42:40.316 DEBUG:teuthology.orchestra.run.smithi156:> sudo shutdown -r now 2024-05-26T11:43:10.319 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-26T11:43:10.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:43:10.320 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:43:28.801 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-05-26T11:43:37.802 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:43:37.803 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:43:40.862 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-05-26T11:43:52.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:43:52.863 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:44:11.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.156 2024-05-26T11:44:26.295 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:44:26.296 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:44:26.812 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-05-26T11:44:27.295 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:44:27.295 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-447.el9.x86_64"... 2024-05-26T11:44:27.295 DEBUG:teuthology.orchestra.run.smithi156:> uname -r 2024-05-26T11:44:27.310 INFO:teuthology.orchestra.run.smithi156.stdout:5.14.0-447.el9.x86_64 2024-05-26T11:44:27.310 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-447.el9.x86_64 vs 5.14.0-447.el9.x86_64 2024-05-26T11:44:27.310 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-26T11:44:27.311 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-26T11:44:28.311 DEBUG:teuthology.parallel:result is None 2024-05-26T11:44:28.312 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-26T11:44:28.318 INFO:teuthology.task.internal:Creating test directory... 2024-05-26T11:44:28.319 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-26T11:44:28.322 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-26T11:44:28.324 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-26T11:44:28.349 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-26T11:44:28.355 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-26T11:44:28.364 INFO:teuthology.task.internal:Creating archive directory... 2024-05-26T11:44:28.365 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-26T11:44:28.392 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-26T11:44:28.394 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-26T11:44:28.496 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-26T11:44:28.502 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-26T11:44:28.502 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-26T11:44:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:44:28.519 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-26T11:44:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:44:28.535 DEBUG:teuthology.orchestra.run.smithi156:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-26T11:44:28.538 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-26T11:44:28.561 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-26T11:44:28.597 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.616 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.620 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.629 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.638 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.665 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-26T11:44:28.668 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-26T11:44:28.675 INFO:teuthology.task.internal:Configuring sudo... 2024-05-26T11:44:28.675 DEBUG:teuthology.orchestra.run.smithi156:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-26T11:44:28.678 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-26T11:44:28.680 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-26T11:44:28.718 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-26T11:44:28.726 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-26T11:44:28.727 DEBUG:teuthology.orchestra.run.smithi156:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-26T11:44:28.761 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-26T11:44:28.764 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-26T11:44:28.783 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:28.879 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:28.949 DEBUG:teuthology.orchestra.run.smithi156:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.000 DEBUG:teuthology.orchestra.run.smithi156:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.057 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:44:29.057 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-26T11:44:29.126 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:29.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:29.223 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.307 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.349 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:44:29.350 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-26T11:44:29.416 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:29.459 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-26T11:44:29.514 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-26T11:44:29.639 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:44:29.639 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-26T11:44:29.704 DEBUG:teuthology.orchestra.run.smithi156:> sudo service rsyslog restart 2024-05-26T11:44:29.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2024-05-26T11:44:29.709 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-05-26T11:44:29.751 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T11:44:29.783 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T11:44:29.827 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T11:44:30.272 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-26T11:44:30.282 INFO:teuthology.task.internal:Starting timer... 2024-05-26T11:44:30.282 INFO:teuthology.run_tasks:Running task pcp... 2024-05-26T11:44:30.301 INFO:teuthology.run_tasks:Running task selinux... 2024-05-26T11:44:30.314 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-26T11:44:30.314 DEBUG:teuthology.orchestra.run.smithi156:> sudo service auditd rotate 2024-05-26T11:44:30.356 INFO:teuthology.orchestra.run.smithi156.stdout:Rotating logs: 2024-05-26T11:44:30.358 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2024-05-26T11:44:30.419 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2024-05-26T11:44:30.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-05-26T11:44:30.481 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-05-26T11:44:30.483 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-26T11:44:30.483 DEBUG:teuthology.orchestra.run.smithi156:> /usr/sbin/getenforce 2024-05-26T11:44:30.513 INFO:teuthology.orchestra.run.smithi156.stdout:Permissive 2024-05-26T11:44:30.514 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2024-05-26T11:44:30.539 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2024-05-26T11:44:30.539 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-05-26T11:44:30.568 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-05-26T11:44:30.568 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi156.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-05-26T11:44:30.569 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T11:44:30.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:44:30.596 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T11:44:30.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:44:30.623 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T11:44:30.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T11:44:30.650 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-26T11:44:30.650 DEBUG:teuthology.orchestra.run.smithi156:> sudo /usr/sbin/setenforce permissive 2024-05-26T11:44:30.680 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2024-05-26T11:44:30.707 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-05-26T11:44:30.730 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-26T11:44:30.739 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-26T11:44:30.743 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-26T11:44:30.862 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-26T11:44:30.872 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-26T11:44:30.873 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi156.front.sepia.ceph.com,smithi186.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-26T11:49:04.400 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi156.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-05-26T11:49:04.402 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:49:04.403 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi156.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:49:04.500 DEBUG:teuthology.orchestra.run.smithi156:> true 2024-05-26T11:49:04.577 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi156.front.sepia.ceph.com' 2024-05-26T11:49:04.577 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-26T11:49:04.577 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:49:04.651 DEBUG:teuthology.orchestra.run.smithi186:> true 2024-05-26T11:49:04.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2024-05-26T11:49:04.738 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-05-26T11:49:04.738 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-26T11:49:04.812 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-05-26T11:49:04.898 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-05-26T11:49:04.899 INFO:teuthology.run_tasks:Running task clock... 2024-05-26T11:49:04.911 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-26T11:49:04.911 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-26T11:49:04.912 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T11:49:04.914 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-26T11:49:04.915 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T11:49:04.917 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-26T11:49:04.917 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T11:49:04.941 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-26T11:49:04.943 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-26T11:49:04.955 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-26T11:49:04.956 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-26T11:49:04.968 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-26T11:49:04.979 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2024-05-26T11:49:04.979 INFO:teuthology.orchestra.run.smithi156.stderr:sudo: ntpd: command not found 2024-05-26T11:49:04.982 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-26T11:49:04.989 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2024-05-26T11:49:04.991 INFO:teuthology.orchestra.run.smithi156.stdout:506 Cannot talk to daemon 2024-05-26T11:49:05.002 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-26T11:49:05.005 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-26T11:49:05.007 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-05-26T11:49:05.015 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-26T11:49:05.017 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-05-26T11:49:05.019 INFO:teuthology.orchestra.run.smithi156.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-26T11:49:05.031 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-26T11:49:05.044 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-26T11:49:05.072 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.076 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.079 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.083 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.101 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-05-26T11:49:05.104 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T11:49:05.104 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-05-26T11:49:05.105 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.105 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.105 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.105 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T11:49:05.106 INFO:teuthology.run_tasks:Running task install... 2024-05-26T11:49:05.113 DEBUG:teuthology.task.install:project ceph 2024-05-26T11:49:05.114 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209'}} 2024-05-26T11:49:05.114 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209'} 2024-05-26T11:49:05.114 INFO:teuthology.task.install:Using flavor: default 2024-05-26T11:49:05.119 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-26T11:49:05.119 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-26T11:49:05.119 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', 'tag': None, 'wait_for_package': False} 2024-05-26T11:49:05.120 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:49:05.122 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', 'tag': None, 'wait_for_package': False} 2024-05-26T11:49:05.122 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:49:05.123 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', 'tag': None, 'wait_for_package': False} 2024-05-26T11:49:05.123 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:49:05.270 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/ 2024-05-26T11:49:05.270 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3867.g9a4c5853 2024-05-26T11:49:05.340 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/ 2024-05-26T11:49:05.340 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3867.g9a4c5853 2024-05-26T11:49:05.361 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/ 2024-05-26T11:49:05.361 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3867.g9a4c5853 2024-05-26T11:49:05.383 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-26T11:49:05.383 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:49:05.383 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-26T11:49:05.408 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-26T11:49:05.408 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-26T11:49:05.450 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-26T11:49:05.451 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:49:05.451 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-26T11:49:05.469 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-26T11:49:05.469 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:49:05.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-26T11:49:05.475 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-26T11:49:05.477 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-26T11:49:05.478 DEBUG:teuthology.orchestra.run.smithi156:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-26T11:49:05.495 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-26T11:49:05.495 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-26T11:49:05.544 DEBUG:teuthology.orchestra.run.smithi156:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-26T11:49:05.547 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-26T11:49:05.560 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-26T11:49:05.611 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2024-05-26T11:49:05.612 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2024-05-26T11:49:05.619 DEBUG:teuthology.orchestra.run.smithi156:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-26T11:49:05.634 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-26T11:49:05.683 INFO:teuthology.orchestra.run.smithi156.stdout:check_obsoletes = 1 2024-05-26T11:49:05.685 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum clean all 2024-05-26T11:49:05.697 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-05-26T11:49:05.699 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-05-26T11:49:05.837 INFO:teuthology.orchestra.run.smithi186.stdout:45 files removed 2024-05-26T11:49:05.864 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-26T11:49:05.910 INFO:teuthology.orchestra.run.smithi156.stdout:45 files removed 2024-05-26T11:49:05.922 INFO:teuthology.orchestra.run.smithi192.stdout:45 files removed 2024-05-26T11:49:05.937 DEBUG:teuthology.orchestra.run.smithi156:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-26T11:49:05.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-26T11:49:06.530 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 207 kB/s | 83 kB 00:00 2024-05-26T11:49:06.680 INFO:teuthology.orchestra.run.smithi156.stdout:ceph packages for x86_64 178 kB/s | 83 kB 00:00 2024-05-26T11:49:06.687 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 174 kB/s | 83 kB 00:00 2024-05-26T11:49:06.964 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 30 kB/s | 12 kB 00:00 2024-05-26T11:49:07.054 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-05-26T11:49:07.056 INFO:teuthology.orchestra.run.smithi156.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-05-26T11:49:07.315 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-26T11:49:07.415 INFO:teuthology.orchestra.run.smithi156.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-26T11:49:07.463 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-05-26T11:49:08.182 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-26T11:49:08.966 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-05-26T11:49:09.355 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.1 MB 00:01 2024-05-26T11:49:10.020 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-05-26T11:49:10.650 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-05-26T11:49:12.589 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 8.9 MB/s | 19 MB 00:02 2024-05-26T11:49:14.204 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-05-26T11:49:14.692 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-05-26T11:49:15.713 INFO:teuthology.orchestra.run.smithi156.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 17 kB 00:00 2024-05-26T11:49:16.401 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 17 kB 00:00 2024-05-26T11:49:16.565 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-05-26T11:49:17.425 INFO:teuthology.orchestra.run.smithi156.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-26T11:49:18.049 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 17 kB 00:00 2024-05-26T11:49:18.947 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 8.8 MB/s | 21 MB 00:02 2024-05-26T11:49:19.926 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-05-26T11:49:22.615 INFO:teuthology.orchestra.run.smithi156.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-26T11:49:24.078 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-26T11:49:24.816 INFO:teuthology.orchestra.run.smithi156.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:24.816 INFO:teuthology.orchestra.run.smithi156.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:24.827 INFO:teuthology.orchestra.run.smithi156.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-26T11:49:24.828 INFO:teuthology.orchestra.run.smithi156.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-26T11:49:24.828 INFO:teuthology.orchestra.run.smithi156.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-26T11:49:24.829 INFO:teuthology.orchestra.run.smithi156.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-26T11:49:24.831 INFO:teuthology.orchestra.run.smithi156.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-26T11:49:24.833 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-26T11:49:24.834 INFO:teuthology.orchestra.run.smithi156.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-26T11:49:24.895 INFO:teuthology.orchestra.run.smithi156.stdout:Dependencies resolved. 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout: Package Arch Version Repository Size 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout:Installing: 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-26T11:49:24.915 INFO:teuthology.orchestra.run.smithi156.stdout: ceph x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 6.6 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 891 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 145 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 1.5 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 143 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 3.5 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 7.4 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 49 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 11 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 49 M 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 273 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 763 k 2024-05-26T11:49:24.916 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 225 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 27 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 18 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 32 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 747 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 127 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-26T11:49:24.917 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 161 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 321 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 300 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 99 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 85 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 171 k 2024-05-26T11:49:24.918 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrading: 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: librados2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.4 M 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-26T11:49:24.919 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout:Installing dependencies: 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 21 M 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 34 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 2.4 M 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 274 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 4.7 M 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 17 M 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 16 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 25 k 2024-05-26T11:49:24.920 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 163 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-26T11:49:24.921 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 501 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-26T11:49:24.922 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 45 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 136 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-26T11:49:24.923 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-26T11:49:24.924 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-26T11:49:24.925 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-26T11:49:24.926 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-26T11:49:24.927 INFO:teuthology.orchestra.run.smithi156.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout:Installing weak dependencies: 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction Summary 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout:======================================================================================= 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout:Install 148 Packages 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout:Upgrade 11 Packages 2024-05-26T11:49:24.928 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:49:24.929 INFO:teuthology.orchestra.run.smithi156.stdout:Total download size: 227 M 2024-05-26T11:49:24.929 INFO:teuthology.orchestra.run.smithi156.stdout:Downloading Packages: 2024-05-26T11:49:25.184 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-05-26T11:49:25.782 INFO:teuthology.orchestra.run.smithi156.stdout:(1/159): ceph-19.0.0-3867.g9a4c5853.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-05-26T11:49:26.083 INFO:teuthology.orchestra.run.smithi156.stdout:(2/159): ceph-fuse-19.0.0-3867.g9a4c5853.el9.x8 2.9 MB/s | 891 kB 00:00 2024-05-26T11:49:26.150 INFO:teuthology.orchestra.run.smithi156.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 2.1 MB/s | 145 kB 00:00 2024-05-26T11:49:26.229 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:26.230 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:26.240 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-26T11:49:26.241 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-26T11:49:26.242 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-26T11:49:26.243 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-26T11:49:26.244 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-26T11:49:26.246 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-26T11:49:26.247 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-26T11:49:26.309 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2024-05-26T11:49:26.319 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 6.6 k 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 891 k 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 145 k 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 1.5 M 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 143 k 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 3.5 M 2024-05-26T11:49:26.320 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 7.4 M 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 49 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 11 M 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 49 M 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 273 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 763 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 225 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 27 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 18 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-26T11:49:26.321 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 32 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 747 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 127 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 161 k 2024-05-26T11:49:26.322 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 321 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 300 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 99 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 85 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 171 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-26T11:49:26.323 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.4 M 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-26T11:49:26.324 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 21 M 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 34 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 2.4 M 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 274 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 4.7 M 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 17 M 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 16 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 25 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-26T11:49:26.325 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 163 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-26T11:49:26.326 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 501 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-26T11:49:26.327 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 45 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 136 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-26T11:49:26.328 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-26T11:49:26.329 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-26T11:49:26.330 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-26T11:49:26.331 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-26T11:49:26.332 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-26T11:49:26.333 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-26T11:49:26.334 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:======================================================================================= 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:Install 148 Packages 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 11 Packages 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 227 M 2024-05-26T11:49:26.335 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2024-05-26T11:49:26.500 INFO:teuthology.orchestra.run.smithi156.stdout:(4/159): ceph-mds-19.0.0-3867.g9a4c5853.el9.x86 6.8 MB/s | 2.4 MB 00:00 2024-05-26T11:49:26.692 INFO:teuthology.orchestra.run.smithi156.stdout:(5/159): ceph-mgr-19.0.0-3867.g9a4c5853.el9.x86 7.9 MB/s | 1.5 MB 00:00 2024-05-26T11:49:27.242 INFO:teuthology.orchestra.run.smithi156.stdout:(6/159): ceph-mon-19.0.0-3867.g9a4c5853.el9.x86 8.5 MB/s | 4.7 MB 00:00 2024-05-26T11:49:27.254 INFO:teuthology.orchestra.run.smithi186.stdout:(1/159): ceph-19.0.0-3867.g9a4c5853.el9.x86_64. 20 kB/s | 6.6 kB 00:00 2024-05-26T11:49:27.360 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:27.361 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-26T11:49:27.372 INFO:teuthology.orchestra.run.smithi192.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-26T11:49:27.373 INFO:teuthology.orchestra.run.smithi192.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-26T11:49:27.374 INFO:teuthology.orchestra.run.smithi192.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-26T11:49:27.375 INFO:teuthology.orchestra.run.smithi192.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-26T11:49:27.376 INFO:teuthology.orchestra.run.smithi192.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-26T11:49:27.379 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-26T11:49:27.380 INFO:teuthology.orchestra.run.smithi192.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-26T11:49:27.384 INFO:teuthology.orchestra.run.smithi156.stdout:(7/159): ceph-base-19.0.0-3867.g9a4c5853.el9.x8 3.0 MB/s | 5.4 MB 00:01 2024-05-26T11:49:27.443 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-05-26T11:49:27.453 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-05-26T11:49:27.453 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-05-26T11:49:27.453 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-05-26T11:49:27.453 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-05-26T11:49:27.453 INFO:teuthology.orchestra.run.smithi192.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 6.6 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 891 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 145 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 1.5 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 143 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 3.5 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 7.4 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 49 k 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 11 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 49 M 2024-05-26T11:49:27.454 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 273 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 763 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 225 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 27 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 18 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 32 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 747 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 127 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-26T11:49:27.455 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 161 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 321 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 300 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 99 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 85 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 171 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-26T11:49:27.456 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager x86_64 1:1.47.90-1.el9 baseos 2.3 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm x86_64 1:1.47.90-1.el9 baseos 1.8 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team x86_64 1:1.47.90-1.el9 baseos 41 k 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui x86_64 1:1.47.90-1.el9 baseos 248 k 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.4 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 3.1 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-05-26T11:49:27.457 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 21 M 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 34 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 2.4 M 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 274 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 4.7 M 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 17 M 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3867.g9a4c5853.el9 ceph-noarch 16 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 25 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-26T11:49:27.458 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 163 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 501 k 2024-05-26T11:49:27.459 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 5.4 M 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-26T11:49:27.460 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 45 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:19.0.0-3867.g9a4c5853.el9 ceph 136 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-05-26T11:49:27.461 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-26T11:49:27.462 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-26T11:49:27.463 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-26T11:49:27.464 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-26T11:49:27.465 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout:Installing weak dependencies: 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout:Install 148 Packages 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 11 Packages 2024-05-26T11:49:27.466 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:49:27.467 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 227 M 2024-05-26T11:49:27.467 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-05-26T11:49:27.918 INFO:teuthology.orchestra.run.smithi156.stdout:(8/159): ceph-common-19.0.0-3867.g9a4c5853.el9. 9.2 MB/s | 21 MB 00:02 2024-05-26T11:49:28.002 INFO:teuthology.orchestra.run.smithi156.stdout:(9/159): ceph-selinux-19.0.0-3867.g9a4c5853.el9 302 kB/s | 25 kB 00:00 2024-05-26T11:49:28.229 INFO:teuthology.orchestra.run.smithi186.stdout:(2/159): ceph-fuse-19.0.0-3867.g9a4c5853.el9.x8 914 kB/s | 891 kB 00:00 2024-05-26T11:49:28.430 INFO:teuthology.orchestra.run.smithi186.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 724 kB/s | 145 kB 00:00 2024-05-26T11:49:28.435 INFO:teuthology.orchestra.run.smithi192.stdout:(1/159): ceph-19.0.0-3867.g9a4c5853.el9.x86_64. 19 kB/s | 6.6 kB 00:00 2024-05-26T11:49:28.685 INFO:teuthology.orchestra.run.smithi156.stdout:(10/159): ceph-osd-19.0.0-3867.g9a4c5853.el9.x8 12 MB/s | 17 MB 00:01 2024-05-26T11:49:28.760 INFO:teuthology.orchestra.run.smithi156.stdout:(11/159): cephfs-mirror-19.0.0-3867.g9a4c5853.e 2.9 MB/s | 225 kB 00:00 2024-05-26T11:49:28.811 INFO:teuthology.orchestra.run.smithi156.stdout:(12/159): cephfs-shell-19.0.0-3867.g9a4c5853.el 541 kB/s | 27 kB 00:00 2024-05-26T11:49:28.822 INFO:teuthology.orchestra.run.smithi186.stdout:(4/159): ceph-base-19.0.0-3867.g9a4c5853.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-05-26T11:49:28.861 INFO:teuthology.orchestra.run.smithi156.stdout:(13/159): libcephfs-devel-19.0.0-3867.g9a4c5853 635 kB/s | 32 kB 00:00 2024-05-26T11:49:28.977 INFO:teuthology.orchestra.run.smithi192.stdout:(2/159): ceph-fuse-19.0.0-3867.g9a4c5853.el9.x8 1.6 MB/s | 891 kB 00:00 2024-05-26T11:49:29.061 INFO:teuthology.orchestra.run.smithi156.stdout:(14/159): libcephfs2-19.0.0-3867.g9a4c5853.el9. 3.6 MB/s | 747 kB 00:00 2024-05-26T11:49:29.088 INFO:teuthology.orchestra.run.smithi186.stdout:(5/159): ceph-mgr-19.0.0-3867.g9a4c5853.el9.x86 5.7 MB/s | 1.5 MB 00:00 2024-05-26T11:49:29.094 INFO:teuthology.orchestra.run.smithi192.stdout:(3/159): ceph-immutable-object-cache-19.0.0-386 1.2 MB/s | 145 kB 00:00 2024-05-26T11:49:29.120 INFO:teuthology.orchestra.run.smithi156.stdout:(15/159): libcephsqlite-19.0.0-3867.g9a4c5853.e 2.7 MB/s | 163 kB 00:00 2024-05-26T11:49:29.237 INFO:teuthology.orchestra.run.smithi156.stdout:(16/159): ceph-radosgw-19.0.0-3867.g9a4c5853.el 5.7 MB/s | 11 MB 00:01 2024-05-26T11:49:29.262 INFO:teuthology.orchestra.run.smithi156.stdout:(17/159): librados-devel-19.0.0-3867.g9a4c5853. 893 kB/s | 127 kB 00:00 2024-05-26T11:49:29.413 INFO:teuthology.orchestra.run.smithi156.stdout:(18/159): libradosstriper1-19.0.0-3867.g9a4c585 2.8 MB/s | 501 kB 00:00 2024-05-26T11:49:29.521 INFO:teuthology.orchestra.run.smithi156.stdout:(19/159): python3-ceph-argparse-19.0.0-3867.g9a 417 kB/s | 45 kB 00:00 2024-05-26T11:49:29.629 INFO:teuthology.orchestra.run.smithi156.stdout:(20/159): python3-ceph-common-19.0.0-3867.g9a4c 1.2 MB/s | 136 kB 00:00 2024-05-26T11:49:29.747 INFO:teuthology.orchestra.run.smithi156.stdout:(21/159): python3-cephfs-19.0.0-3867.g9a4c5853. 1.4 MB/s | 161 kB 00:00 2024-05-26T11:49:29.814 INFO:teuthology.orchestra.run.smithi186.stdout:(6/159): ceph-mon-19.0.0-3867.g9a4c5853.el9.x86 6.5 MB/s | 4.7 MB 00:00 2024-05-26T11:49:29.855 INFO:teuthology.orchestra.run.smithi156.stdout:(22/159): python3-rados-19.0.0-3867.g9a4c5853.e 2.9 MB/s | 321 kB 00:00 2024-05-26T11:49:29.980 INFO:teuthology.orchestra.run.smithi156.stdout:(23/159): python3-rbd-19.0.0-3867.g9a4c5853.el9 2.3 MB/s | 300 kB 00:00 2024-05-26T11:49:30.053 INFO:teuthology.orchestra.run.smithi192.stdout:(4/159): ceph-mds-19.0.0-3867.g9a4c5853.el9.x86 2.5 MB/s | 2.4 MB 00:00 2024-05-26T11:49:30.131 INFO:teuthology.orchestra.run.smithi156.stdout:(24/159): librgw2-19.0.0-3867.g9a4c5853.el9.x86 6.2 MB/s | 5.4 MB 00:00 2024-05-26T11:49:30.148 INFO:teuthology.orchestra.run.smithi156.stdout:(25/159): python3-rgw-19.0.0-3867.g9a4c5853.el9 594 kB/s | 99 kB 00:00 2024-05-26T11:49:30.198 INFO:teuthology.orchestra.run.smithi156.stdout:(26/159): rbd-fuse-19.0.0-3867.g9a4c5853.el9.x8 1.2 MB/s | 85 kB 00:00 2024-05-26T11:49:30.282 INFO:teuthology.orchestra.run.smithi156.stdout:(27/159): rbd-nbd-19.0.0-3867.g9a4c5853.el9.x86 2.0 MB/s | 171 kB 00:00 2024-05-26T11:49:30.407 INFO:teuthology.orchestra.run.smithi156.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3867.g 273 kB/s | 34 kB 00:00 2024-05-26T11:49:30.419 INFO:teuthology.orchestra.run.smithi192.stdout:(5/159): ceph-mgr-19.0.0-3867.g9a4c5853.el9.x86 4.1 MB/s | 1.5 MB 00:00 2024-05-26T11:49:30.455 INFO:teuthology.orchestra.run.smithi186.stdout:(7/159): ceph-mds-19.0.0-3867.g9a4c5853.el9.x86 1.2 MB/s | 2.4 MB 00:02 2024-05-26T11:49:30.465 INFO:teuthology.orchestra.run.smithi156.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3867.g9a4c585 2.4 MB/s | 143 kB 00:00 2024-05-26T11:49:30.816 INFO:teuthology.orchestra.run.smithi156.stdout:(30/159): rbd-mirror-19.0.0-3867.g9a4c5853.el9. 4.6 MB/s | 3.1 MB 00:00 2024-05-26T11:49:31.208 INFO:teuthology.orchestra.run.smithi156.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3867.g9a4c5 4.8 MB/s | 3.5 MB 00:00 2024-05-26T11:49:31.308 INFO:teuthology.orchestra.run.smithi156.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3867.g9a 2.7 MB/s | 274 kB 00:00 2024-05-26T11:49:31.400 INFO:teuthology.orchestra.run.smithi156.stdout:(33/159): ceph-mgr-rook-19.0.0-3867.g9a4c5853.e 537 kB/s | 49 kB 00:00 2024-05-26T11:49:31.525 INFO:teuthology.orchestra.run.smithi156.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3867.g9 132 kB/s | 16 kB 00:00 2024-05-26T11:49:31.609 INFO:teuthology.orchestra.run.smithi156.stdout:(35/159): ceph-volume-19.0.0-3867.g9a4c5853.el9 3.2 MB/s | 273 kB 00:00 2024-05-26T11:49:31.801 INFO:teuthology.orchestra.run.smithi156.stdout:(36/159): cephadm-19.0.0-3867.g9a4c5853.el9.noa 3.9 MB/s | 763 kB 00:00 2024-05-26T11:49:31.828 INFO:teuthology.orchestra.run.smithi192.stdout:(6/159): ceph-mon-19.0.0-3867.g9a4c5853.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-05-26T11:49:31.868 INFO:teuthology.orchestra.run.smithi156.stdout:(37/159): cephfs-top-19.0.0-3867.g9a4c5853.el9. 272 kB/s | 18 kB 00:00 2024-05-26T11:49:31.903 INFO:teuthology.orchestra.run.smithi192.stdout:(7/159): ceph-base-19.0.0-3867.g9a4c5853.el9.x8 1.4 MB/s | 5.4 MB 00:03 2024-05-26T11:49:32.385 INFO:teuthology.orchestra.run.smithi156.stdout:(38/159): ceph-mgr-diskprediction-local-19.0.0- 4.7 MB/s | 7.4 MB 00:01 2024-05-26T11:49:32.586 INFO:teuthology.orchestra.run.smithi156.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 112 kB/s | 80 kB 00:00 2024-05-26T11:49:32.914 INFO:teuthology.orchestra.run.smithi186.stdout:(8/159): ceph-common-19.0.0-3867.g9a4c5853.el9. 3.5 MB/s | 21 MB 00:05 2024-05-26T11:49:33.089 INFO:teuthology.orchestra.run.smithi186.stdout:(9/159): ceph-osd-19.0.0-3867.g9a4c5853.el9.x86 5.1 MB/s | 17 MB 00:03 2024-05-26T11:49:33.114 INFO:teuthology.orchestra.run.smithi186.stdout:(10/159): ceph-selinux-19.0.0-3867.g9a4c5853.el 126 kB/s | 25 kB 00:00 2024-05-26T11:49:33.249 INFO:teuthology.orchestra.run.smithi186.stdout:(11/159): cephfs-mirror-19.0.0-3867.g9a4c5853.e 1.6 MB/s | 225 kB 00:00 2024-05-26T11:49:33.323 INFO:teuthology.orchestra.run.smithi186.stdout:(12/159): cephfs-shell-19.0.0-3867.g9a4c5853.el 360 kB/s | 27 kB 00:00 2024-05-26T11:49:33.377 INFO:teuthology.orchestra.run.smithi156.stdout:(40/159): libgfortran-11.4.1-3.el9.x86_64.rpm 810 kB/s | 803 kB 00:00 2024-05-26T11:49:33.402 INFO:teuthology.orchestra.run.smithi156.stdout:(41/159): libquadmath-11.4.1-3.el9.x86_64.rpm 239 kB/s | 195 kB 00:00 2024-05-26T11:49:33.407 INFO:teuthology.orchestra.run.smithi186.stdout:(13/159): libcephfs-devel-19.0.0-3867.g9a4c5853 381 kB/s | 32 kB 00:00 2024-05-26T11:49:33.582 INFO:teuthology.orchestra.run.smithi186.stdout:(14/159): libcephfs2-19.0.0-3867.g9a4c5853.el9. 4.2 MB/s | 747 kB 00:00 2024-05-26T11:49:33.674 INFO:teuthology.orchestra.run.smithi186.stdout:(15/159): libcephsqlite-19.0.0-3867.g9a4c5853.e 1.7 MB/s | 163 kB 00:00 2024-05-26T11:49:33.757 INFO:teuthology.orchestra.run.smithi186.stdout:(16/159): librados-devel-19.0.0-3867.g9a4c5853. 1.5 MB/s | 127 kB 00:00 2024-05-26T11:49:33.916 INFO:teuthology.orchestra.run.smithi186.stdout:(17/159): libradosstriper1-19.0.0-3867.g9a4c585 3.1 MB/s | 501 kB 00:00 2024-05-26T11:49:34.203 INFO:teuthology.orchestra.run.smithi156.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 307 kB/s | 253 kB 00:00 2024-05-26T11:49:34.420 INFO:teuthology.orchestra.run.smithi156.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2024-05-26T11:49:34.537 INFO:teuthology.orchestra.run.smithi192.stdout:(8/159): ceph-osd-19.0.0-3867.g9a4c5853.el9.x86 6.2 MB/s | 17 MB 00:02 2024-05-26T11:49:34.604 INFO:teuthology.orchestra.run.smithi192.stdout:(9/159): ceph-selinux-19.0.0-3867.g9a4c5853.el9 379 kB/s | 25 kB 00:00 2024-05-26T11:49:34.816 INFO:teuthology.orchestra.run.smithi186.stdout:(18/159): librgw2-19.0.0-3867.g9a4c5853.el9.x86 6.0 MB/s | 5.4 MB 00:00 2024-05-26T11:49:34.892 INFO:teuthology.orchestra.run.smithi186.stdout:(19/159): python3-ceph-argparse-19.0.0-3867.g9a 605 kB/s | 45 kB 00:00 2024-05-26T11:49:35.000 INFO:teuthology.orchestra.run.smithi186.stdout:(20/159): python3-ceph-common-19.0.0-3867.g9a4c 1.2 MB/s | 136 kB 00:00 2024-05-26T11:49:35.011 INFO:teuthology.orchestra.run.smithi156.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 132 kB/s | 106 kB 00:00 2024-05-26T11:49:35.092 INFO:teuthology.orchestra.run.smithi186.stdout:(21/159): python3-cephfs-19.0.0-3867.g9a4c5853. 1.7 MB/s | 161 kB 00:00 2024-05-26T11:49:35.176 INFO:teuthology.orchestra.run.smithi186.stdout:(22/159): python3-rados-19.0.0-3867.g9a4c5853.e 3.8 MB/s | 321 kB 00:00 2024-05-26T11:49:35.203 INFO:teuthology.orchestra.run.smithi156.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 173 kB/s | 135 kB 00:00 2024-05-26T11:49:35.259 INFO:teuthology.orchestra.run.smithi186.stdout:(23/159): python3-rbd-19.0.0-3867.g9a4c5853.el9 3.5 MB/s | 300 kB 00:00 2024-05-26T11:49:35.321 INFO:teuthology.orchestra.run.smithi192.stdout:(10/159): ceph-common-19.0.0-3867.g9a4c5853.el9 2.9 MB/s | 21 MB 00:07 2024-05-26T11:49:35.326 INFO:teuthology.orchestra.run.smithi186.stdout:(24/159): python3-rgw-19.0.0-3867.g9a4c5853.el9 1.4 MB/s | 99 kB 00:00 2024-05-26T11:49:35.418 INFO:teuthology.orchestra.run.smithi186.stdout:(25/159): rbd-fuse-19.0.0-3867.g9a4c5853.el9.x8 933 kB/s | 85 kB 00:00 2024-05-26T11:49:35.462 INFO:teuthology.orchestra.run.smithi192.stdout:(11/159): cephfs-mirror-19.0.0-3867.g9a4c5853.e 1.6 MB/s | 225 kB 00:00 2024-05-26T11:49:35.563 INFO:teuthology.orchestra.run.smithi192.stdout:(12/159): cephfs-shell-19.0.0-3867.g9a4c5853.el 270 kB/s | 27 kB 00:00 2024-05-26T11:49:35.663 INFO:teuthology.orchestra.run.smithi192.stdout:(13/159): libcephfs-devel-19.0.0-3867.g9a4c5853 318 kB/s | 32 kB 00:00 2024-05-26T11:49:35.787 INFO:teuthology.orchestra.run.smithi156.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 161 kB/s | 125 kB 00:00 2024-05-26T11:49:35.913 INFO:teuthology.orchestra.run.smithi192.stdout:(14/159): libcephfs2-19.0.0-3867.g9a4c5853.el9. 2.9 MB/s | 747 kB 00:00 2024-05-26T11:49:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:(15/159): libcephsqlite-19.0.0-3867.g9a4c5853.e 1.9 MB/s | 163 kB 00:00 2024-05-26T11:49:36.029 INFO:teuthology.orchestra.run.smithi156.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 260 kB/s | 215 kB 00:00 2024-05-26T11:49:36.035 INFO:teuthology.orchestra.run.smithi186.stdout:(26/159): rbd-mirror-19.0.0-3867.g9a4c5853.el9. 5.0 MB/s | 3.1 MB 00:00 2024-05-26T11:49:36.072 INFO:teuthology.orchestra.run.smithi192.stdout:(16/159): librados-devel-19.0.0-3867.g9a4c5853. 1.6 MB/s | 127 kB 00:00 2024-05-26T11:49:36.127 INFO:teuthology.orchestra.run.smithi186.stdout:(27/159): rbd-nbd-19.0.0-3867.g9a4c5853.el9.x86 1.8 MB/s | 171 kB 00:00 2024-05-26T11:49:36.211 INFO:teuthology.orchestra.run.smithi186.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3867.g 412 kB/s | 34 kB 00:00 2024-05-26T11:49:36.231 INFO:teuthology.orchestra.run.smithi192.stdout:(17/159): libradosstriper1-19.0.0-3867.g9a4c585 3.1 MB/s | 501 kB 00:00 2024-05-26T11:49:36.294 INFO:teuthology.orchestra.run.smithi186.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3867.g9a4c585 1.7 MB/s | 143 kB 00:00 2024-05-26T11:49:36.732 INFO:teuthology.orchestra.run.smithi192.stdout:(18/159): ceph-radosgw-19.0.0-3867.g9a4c5853.el 2.2 MB/s | 11 MB 00:04 2024-05-26T11:49:36.746 INFO:teuthology.orchestra.run.smithi156.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 719 kB/s | 689 kB 00:00 2024-05-26T11:49:36.832 INFO:teuthology.orchestra.run.smithi192.stdout:(19/159): python3-ceph-argparse-19.0.0-3867.g9a 453 kB/s | 45 kB 00:00 2024-05-26T11:49:36.932 INFO:teuthology.orchestra.run.smithi192.stdout:(20/159): python3-ceph-common-19.0.0-3867.g9a4c 1.3 MB/s | 136 kB 00:00 2024-05-26T11:49:36.978 INFO:teuthology.orchestra.run.smithi186.stdout:(30/159): ceph-mgr-dashboard-19.0.0-3867.g9a4c5 5.2 MB/s | 3.5 MB 00:00 2024-05-26T11:49:37.016 INFO:teuthology.orchestra.run.smithi192.stdout:(21/159): python3-cephfs-19.0.0-3867.g9a4c5853. 1.9 MB/s | 161 kB 00:00 2024-05-26T11:49:37.029 INFO:teuthology.orchestra.run.smithi156.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 969 kB/s | 943 kB 00:00 2024-05-26T11:49:37.199 INFO:teuthology.orchestra.run.smithi192.stdout:(22/159): python3-rados-19.0.0-3867.g9a4c5853.e 1.7 MB/s | 321 kB 00:00 2024-05-26T11:49:37.203 INFO:teuthology.orchestra.run.smithi186.stdout:(31/159): ceph-radosgw-19.0.0-3867.g9a4c5853.el 1.6 MB/s | 11 MB 00:06 2024-05-26T11:49:37.358 INFO:teuthology.orchestra.run.smithi192.stdout:(23/159): python3-rbd-19.0.0-3867.g9a4c5853.el9 1.9 MB/s | 300 kB 00:00 2024-05-26T11:49:37.395 INFO:teuthology.orchestra.run.smithi186.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3867.g9a 1.4 MB/s | 274 kB 00:00 2024-05-26T11:49:37.450 INFO:teuthology.orchestra.run.smithi192.stdout:(24/159): python3-rgw-19.0.0-3867.g9a4c5853.el9 1.1 MB/s | 99 kB 00:00 2024-05-26T11:49:37.471 INFO:teuthology.orchestra.run.smithi156.stdout:(50/159): boost-program-options-1.75.0-8.el9.x8 147 kB/s | 107 kB 00:00 2024-05-26T11:49:37.495 INFO:teuthology.orchestra.run.smithi186.stdout:(33/159): ceph-mgr-rook-19.0.0-3867.g9a4c5853.e 494 kB/s | 49 kB 00:00 2024-05-26T11:49:37.542 INFO:teuthology.orchestra.run.smithi192.stdout:(25/159): rbd-fuse-19.0.0-3867.g9a4c5853.el9.x8 931 kB/s | 85 kB 00:00 2024-05-26T11:49:37.587 INFO:teuthology.orchestra.run.smithi186.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3867.g9 179 kB/s | 16 kB 00:00 2024-05-26T11:49:37.596 INFO:teuthology.orchestra.run.smithi156.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 38 kB/s | 22 kB 00:00 2024-05-26T11:49:37.771 INFO:teuthology.orchestra.run.smithi186.stdout:(35/159): ceph-volume-19.0.0-3867.g9a4c5853.el9 1.5 MB/s | 273 kB 00:00 2024-05-26T11:49:37.792 INFO:teuthology.orchestra.run.smithi192.stdout:(26/159): librgw2-19.0.0-3867.g9a4c5853.el9.x86 3.4 MB/s | 5.4 MB 00:01 2024-05-26T11:49:37.892 INFO:teuthology.orchestra.run.smithi192.stdout:(27/159): rbd-nbd-19.0.0-3867.g9a4c5853.el9.x86 1.7 MB/s | 171 kB 00:00 2024-05-26T11:49:37.984 INFO:teuthology.orchestra.run.smithi192.stdout:(28/159): ceph-grafana-dashboards-19.0.0-3867.g 373 kB/s | 34 kB 00:00 2024-05-26T11:49:38.084 INFO:teuthology.orchestra.run.smithi192.stdout:(29/159): ceph-mgr-cephadm-19.0.0-3867.g9a4c585 1.4 MB/s | 143 kB 00:00 2024-05-26T11:49:38.205 INFO:teuthology.orchestra.run.smithi186.stdout:(36/159): cephadm-19.0.0-3867.g9a4c5853.el9.noa 1.7 MB/s | 763 kB 00:00 2024-05-26T11:49:38.213 INFO:teuthology.orchestra.run.smithi156.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-05-26T11:49:38.305 INFO:teuthology.orchestra.run.smithi186.stdout:(37/159): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-05-26T11:49:38.305 INFO:teuthology.orchestra.run.smithi156.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 376 kB/s | 313 kB 00:00 2024-05-26T11:49:38.322 INFO:teuthology.orchestra.run.smithi186.stdout:(38/159): cephfs-top-19.0.0-3867.g9a4c5853.el9. 156 kB/s | 18 kB 00:00 2024-05-26T11:49:38.430 INFO:teuthology.orchestra.run.smithi186.stdout:(39/159): fuse-2.9.9-15.el9.x86_64.rpm 643 kB/s | 80 kB 00:00 2024-05-26T11:49:38.514 INFO:teuthology.orchestra.run.smithi186.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-26T11:49:38.539 INFO:teuthology.orchestra.run.smithi186.stdout:(41/159): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-05-26T11:49:38.597 INFO:teuthology.orchestra.run.smithi186.stdout:(42/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.0 MB/s | 253 kB 00:00 2024-05-26T11:49:38.640 INFO:teuthology.orchestra.run.smithi186.stdout:(43/159): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.2 MB 00:00 2024-05-26T11:49:38.665 INFO:teuthology.orchestra.run.smithi186.stdout:(44/159): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 106 kB 00:00 2024-05-26T11:49:38.690 INFO:teuthology.orchestra.run.smithi186.stdout:(45/159): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-05-26T11:49:38.716 INFO:teuthology.orchestra.run.smithi186.stdout:(46/159): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-26T11:49:38.741 INFO:teuthology.orchestra.run.smithi186.stdout:(47/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-26T11:49:38.776 INFO:teuthology.orchestra.run.smithi192.stdout:(30/159): rbd-mirror-19.0.0-3867.g9a4c5853.el9. 2.5 MB/s | 3.1 MB 00:01 2024-05-26T11:49:38.800 INFO:teuthology.orchestra.run.smithi186.stdout:(48/159): automake-1.16.2-8.el9.noarch.rpm 8.1 MB/s | 689 kB 00:00 2024-05-26T11:49:38.833 INFO:teuthology.orchestra.run.smithi186.stdout:(49/159): bison-3.7.4-5.el9.x86_64.rpm 10 MB/s | 943 kB 00:00 2024-05-26T11:49:38.856 INFO:teuthology.orchestra.run.smithi156.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 33 kB/s | 18 kB 00:00 2024-05-26T11:49:38.908 INFO:teuthology.orchestra.run.smithi186.stdout:(50/159): elfutils-libelf-devel-0.191-4.el9.x86 289 kB/s | 22 kB 00:00 2024-05-26T11:49:38.934 INFO:teuthology.orchestra.run.smithi186.stdout:(51/159): boost-program-options-1.75.0-8.el9.x8 799 kB/s | 107 kB 00:00 2024-05-26T11:49:38.968 INFO:teuthology.orchestra.run.smithi192.stdout:(31/159): ceph-mgr-dashboard-19.0.0-3867.g9a4c5 4.0 MB/s | 3.5 MB 00:00 2024-05-26T11:49:38.984 INFO:teuthology.orchestra.run.smithi186.stdout:(52/159): flexiblas-3.0.4-8.el9.x86_64.rpm 656 kB/s | 33 kB 00:00 2024-05-26T11:49:39.023 INFO:teuthology.orchestra.run.smithi186.stdout:(53/159): flex-2.6.4-9.el9.x86_64.rpm 3.0 MB/s | 313 kB 00:00 2024-05-26T11:49:39.068 INFO:teuthology.orchestra.run.smithi186.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 313 kB/s | 18 kB 00:00 2024-05-26T11:49:39.069 INFO:teuthology.orchestra.run.smithi192.stdout:(32/159): ceph-mgr-modules-core-19.0.0-3867.g9a 2.7 MB/s | 274 kB 00:00 2024-05-26T11:49:39.144 INFO:teuthology.orchestra.run.smithi186.stdout:(55/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 19 MB/s | 3.0 MB 00:00 2024-05-26T11:49:39.144 INFO:teuthology.orchestra.run.smithi192.stdout:(33/159): ceph-mgr-rook-19.0.0-3867.g9a4c5853.e 659 kB/s | 49 kB 00:00 2024-05-26T11:49:39.169 INFO:teuthology.orchestra.run.smithi186.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-05-26T11:49:39.219 INFO:teuthology.orchestra.run.smithi192.stdout:(34/159): ceph-prometheus-alerts-19.0.0-3867.g9 220 kB/s | 16 kB 00:00 2024-05-26T11:49:39.236 INFO:teuthology.orchestra.run.smithi186.stdout:(57/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 680 kB/s | 45 kB 00:00 2024-05-26T11:49:39.253 INFO:teuthology.orchestra.run.smithi186.stdout:(58/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.4 MB/s | 160 kB 00:00 2024-05-26T11:49:39.311 INFO:teuthology.orchestra.run.smithi192.stdout:(35/159): ceph-volume-19.0.0-3867.g9a4c5853.el9 2.9 MB/s | 273 kB 00:00 2024-05-26T11:49:39.328 INFO:teuthology.orchestra.run.smithi186.stdout:(59/159): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-05-26T11:49:39.361 INFO:teuthology.orchestra.run.smithi186.stdout:(60/159): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 662 kB 00:00 2024-05-26T11:49:39.372 INFO:teuthology.orchestra.run.smithi156.stdout:(55/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-05-26T11:49:39.387 INFO:teuthology.orchestra.run.smithi186.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 435 kB/s | 25 kB 00:00 2024-05-26T11:49:39.398 INFO:teuthology.orchestra.run.smithi156.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-05-26T11:49:39.437 INFO:teuthology.orchestra.run.smithi186.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 632 kB/s | 47 kB 00:00 2024-05-26T11:49:39.462 INFO:teuthology.orchestra.run.smithi186.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-05-26T11:49:39.520 INFO:teuthology.orchestra.run.smithi192.stdout:(36/159): cephadm-19.0.0-3867.g9a4c5853.el9.noa 3.6 MB/s | 763 kB 00:00 2024-05-26T11:49:39.587 INFO:teuthology.orchestra.run.smithi192.stdout:(37/159): cephfs-top-19.0.0-3867.g9a4c5853.el9. 274 kB/s | 18 kB 00:00 2024-05-26T11:49:39.687 INFO:teuthology.orchestra.run.smithi192.stdout:(38/159): fuse-2.9.9-15.el9.x86_64.rpm 804 kB/s | 80 kB 00:00 2024-05-26T11:49:39.804 INFO:teuthology.orchestra.run.smithi192.stdout:(39/159): libgfortran-11.4.1-3.el9.x86_64.rpm 6.7 MB/s | 803 kB 00:00 2024-05-26T11:49:39.846 INFO:teuthology.orchestra.run.smithi192.stdout:(40/159): libquadmath-11.4.1-3.el9.x86_64.rpm 4.6 MB/s | 195 kB 00:00 2024-05-26T11:49:39.897 INFO:teuthology.orchestra.run.smithi192.stdout:(41/159): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.0 MB/s | 253 kB 00:00 2024-05-26T11:49:39.946 INFO:teuthology.orchestra.run.smithi186.stdout:(64/159): ceph-test-19.0.0-3867.g9a4c5853.el9.x 7.2 MB/s | 49 MB 00:06 2024-05-26T11:49:39.971 INFO:teuthology.orchestra.run.smithi186.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 69 kB/s | 35 kB 00:00 2024-05-26T11:49:39.972 INFO:teuthology.orchestra.run.smithi192.stdout:(42/159): python3-cryptography-36.0.1-4.el9.x86 16 MB/s | 1.2 MB 00:00 2024-05-26T11:49:39.997 INFO:teuthology.orchestra.run.smithi186.stdout:(66/159): lua-5.4.4-4.el9.x86_64.rpm 337 kB/s | 188 kB 00:00 2024-05-26T11:49:40.114 INFO:teuthology.orchestra.run.smithi186.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 31 MB/s | 4.4 MB 00:00 2024-05-26T11:49:40.164 INFO:teuthology.orchestra.run.smithi186.stdout:(68/159): postgresql-13.14-1.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-05-26T11:49:40.206 INFO:teuthology.orchestra.run.smithi192.stdout:(43/159): python3-ply-3.11-14.el9.noarch.rpm 456 kB/s | 106 kB 00:00 2024-05-26T11:49:40.222 INFO:teuthology.orchestra.run.smithi186.stdout:(69/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-26T11:49:40.247 INFO:teuthology.orchestra.run.smithi192.stdout:(44/159): python3-pycparser-2.20-6.el9.noarch.r 3.2 MB/s | 135 kB 00:00 2024-05-26T11:49:40.248 INFO:teuthology.orchestra.run.smithi156.stdout:(57/159): ceph-test-19.0.0-3867.g9a4c5853.el9.x 4.0 MB/s | 49 MB 00:12 2024-05-26T11:49:40.264 INFO:teuthology.orchestra.run.smithi186.stdout:(70/159): postgresql-contrib-13.14-1.el9.x86_64 5.9 MB/s | 911 kB 00:00 2024-05-26T11:49:40.273 INFO:teuthology.orchestra.run.smithi156.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 52 kB/s | 45 kB 00:00 2024-05-26T11:49:40.323 INFO:teuthology.orchestra.run.smithi192.stdout:(45/159): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-05-26T11:49:40.373 INFO:teuthology.orchestra.run.smithi186.stdout:(71/159): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-05-26T11:49:40.373 INFO:teuthology.orchestra.run.smithi192.stdout:(46/159): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-05-26T11:49:40.440 INFO:teuthology.orchestra.run.smithi186.stdout:(72/159): openblas-openmp-0.3.21-2.el9.x86_64.r 9.6 MB/s | 4.7 MB 00:00 2024-05-26T11:49:40.465 INFO:teuthology.orchestra.run.smithi186.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 954 kB/s | 87 kB 00:00 2024-05-26T11:49:40.532 INFO:teuthology.orchestra.run.smithi192.stdout:(47/159): automake-1.16.2-8.el9.noarch.rpm 4.3 MB/s | 689 kB 00:00 2024-05-26T11:49:40.549 INFO:teuthology.orchestra.run.smithi186.stdout:(74/159): postgresql-server-13.14-1.el9.x86_64. 18 MB/s | 5.8 MB 00:00 2024-05-26T11:49:40.549 INFO:teuthology.orchestra.run.smithi156.stdout:(59/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 136 kB/s | 160 kB 00:01 2024-05-26T11:49:40.566 INFO:teuthology.orchestra.run.smithi186.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 610 kB/s | 61 kB 00:00 2024-05-26T11:49:40.599 INFO:teuthology.orchestra.run.smithi192.stdout:(48/159): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 943 kB 00:00 2024-05-26T11:49:40.641 INFO:teuthology.orchestra.run.smithi192.stdout:(49/159): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 107 kB 00:00 2024-05-26T11:49:40.657 INFO:teuthology.orchestra.run.smithi186.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-05-26T11:49:40.683 INFO:teuthology.orchestra.run.smithi186.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-05-26T11:49:40.708 INFO:teuthology.orchestra.run.smithi186.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 1.7 MB/s | 248 kB 00:00 2024-05-26T11:49:40.724 INFO:teuthology.orchestra.run.smithi192.stdout:(50/159): ceph-mgr-diskprediction-local-19.0.0- 3.8 MB/s | 7.4 MB 00:01 2024-05-26T11:49:40.750 INFO:teuthology.orchestra.run.smithi186.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 279 kB/s | 26 kB 00:00 2024-05-26T11:49:40.758 INFO:teuthology.orchestra.run.smithi192.stdout:(51/159): elfutils-libelf-devel-0.191-4.el9.x86 186 kB/s | 22 kB 00:00 2024-05-26T11:49:40.775 INFO:teuthology.orchestra.run.smithi186.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-05-26T11:49:40.792 INFO:teuthology.orchestra.run.smithi186.stdout:(81/159): python3-jsonpointer-2.0-4.el9.noarch. 229 kB/s | 19 kB 00:00 2024-05-26T11:49:40.834 INFO:teuthology.orchestra.run.smithi186.stdout:(82/159): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-05-26T11:49:40.935 INFO:teuthology.orchestra.run.smithi186.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 5.1 MB 00:00 2024-05-26T11:49:40.960 INFO:teuthology.orchestra.run.smithi186.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-05-26T11:49:40.985 INFO:teuthology.orchestra.run.smithi186.stdout:(85/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.3 MB/s | 250 kB 00:00 2024-05-26T11:49:41.002 INFO:teuthology.orchestra.run.smithi186.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 1.1 MB/s | 77 kB 00:00 2024-05-26T11:49:41.036 INFO:teuthology.orchestra.run.smithi186.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-26T11:49:41.061 INFO:teuthology.orchestra.run.smithi186.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 159 kB 00:00 2024-05-26T11:49:41.078 INFO:teuthology.orchestra.run.smithi186.stdout:(89/159): python3-prettytable-0.7.2-27.el9.noar 360 kB/s | 42 kB 00:00 2024-05-26T11:49:41.103 INFO:teuthology.orchestra.run.smithi186.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 279 kB 00:00 2024-05-26T11:49:41.132 INFO:teuthology.orchestra.run.smithi156.stdout:(60/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 44 kB/s | 25 kB 00:00 2024-05-26T11:49:41.145 INFO:teuthology.orchestra.run.smithi186.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 614 kB/s | 51 kB 00:00 2024-05-26T11:49:41.162 INFO:teuthology.orchestra.run.smithi186.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 640 kB/s | 54 kB 00:00 2024-05-26T11:49:41.166 INFO:teuthology.orchestra.run.smithi156.stdout:(61/159): librdkafka-1.6.1-102.el9.x86_64.rpm 722 kB/s | 662 kB 00:00 2024-05-26T11:49:41.199 INFO:teuthology.orchestra.run.smithi156.stdout:(62/159): libtool-2.4.6-45.el9.x86_64.rpm 629 kB/s | 582 kB 00:00 2024-05-26T11:49:41.200 INFO:teuthology.orchestra.run.smithi192.stdout:(52/159): ceph-test-19.0.0-3867.g9a4c5853.el9.x 7.4 MB/s | 49 MB 00:06 2024-05-26T11:49:41.204 INFO:teuthology.orchestra.run.smithi186.stdout:(93/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-05-26T11:49:41.221 INFO:teuthology.orchestra.run.smithi186.stdout:(94/159): python3-toml-0.10.2-6.el9.noarch.rpm 714 kB/s | 42 kB 00:00 2024-05-26T11:49:41.225 INFO:teuthology.orchestra.run.smithi192.stdout:(53/159): flexiblas-3.0.4-8.el9.x86_64.rpm 70 kB/s | 33 kB 00:00 2024-05-26T11:49:41.254 INFO:teuthology.orchestra.run.smithi186.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-26T11:49:41.271 INFO:teuthology.orchestra.run.smithi186.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 714 kB/s | 36 kB 00:00 2024-05-26T11:49:41.317 INFO:teuthology.orchestra.run.smithi192.stdout:(54/159): flexiblas-openblas-openmp-3.0.4-8.el9 198 kB/s | 18 kB 00:00 2024-05-26T11:49:41.342 INFO:teuthology.orchestra.run.smithi192.stdout:(55/159): flex-2.6.4-9.el9.x86_64.rpm 507 kB/s | 313 kB 00:00 2024-05-26T11:49:41.355 INFO:teuthology.orchestra.run.smithi186.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 472 kB/s | 47 kB 00:00 2024-05-26T11:49:41.368 INFO:teuthology.orchestra.run.smithi192.stdout:(56/159): libaio-devel-0.3.111-13.el9.x86_64.rp 223 kB/s | 11 kB 00:00 2024-05-26T11:49:41.393 INFO:teuthology.orchestra.run.smithi192.stdout:(57/159): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-26T11:49:41.418 INFO:teuthology.orchestra.run.smithi192.stdout:(58/159): librabbitmq-0.11.0-7.el9.x86_64.rpm 903 kB/s | 45 kB 00:00 2024-05-26T11:49:41.460 INFO:teuthology.orchestra.run.smithi192.stdout:(59/159): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 662 kB 00:00 2024-05-26T11:49:41.494 INFO:teuthology.orchestra.run.smithi192.stdout:(60/159): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 582 kB 00:00 2024-05-26T11:49:41.511 INFO:teuthology.orchestra.run.smithi192.stdout:(61/159): libuuid-devel-2.37.4-18.el9.x86_64.rp 507 kB/s | 25 kB 00:00 2024-05-26T11:49:41.513 INFO:teuthology.orchestra.run.smithi186.stdout:(98/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-05-26T11:49:41.528 INFO:teuthology.orchestra.run.smithi192.stdout:(62/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-05-26T11:49:41.539 INFO:teuthology.orchestra.run.smithi186.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-05-26T11:49:41.553 INFO:teuthology.orchestra.run.smithi192.stdout:(63/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.8 MB/s | 292 kB 00:00 2024-05-26T11:49:41.564 INFO:teuthology.orchestra.run.smithi186.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 277 kB/s | 58 kB 00:00 2024-05-26T11:49:41.579 INFO:teuthology.orchestra.run.smithi192.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-26T11:49:41.596 INFO:teuthology.orchestra.run.smithi192.stdout:(65/159): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 35 kB 00:00 2024-05-26T11:49:41.646 INFO:teuthology.orchestra.run.smithi192.stdout:(66/159): flexiblas-netlib-3.0.4-8.el9.x86_64.r 6.7 MB/s | 3.0 MB 00:00 2024-05-26T11:49:41.664 INFO:teuthology.orchestra.run.smithi186.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 624 kB/s | 94 kB 00:00 2024-05-26T11:49:41.698 INFO:teuthology.orchestra.run.smithi186.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 336 kB/s | 45 kB 00:00 2024-05-26T11:49:41.713 INFO:teuthology.orchestra.run.smithi192.stdout:(67/159): postgresql-13.14-1.el9.x86_64.rpm 24 MB/s | 1.6 MB 00:00 2024-05-26T11:49:41.723 INFO:teuthology.orchestra.run.smithi186.stdout:(103/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 401 kB/s | 64 kB 00:00 2024-05-26T11:49:41.758 INFO:teuthology.orchestra.run.smithi156.stdout:(63/159): libzstd-devel-1.5.1-2.el9.x86_64.rpm 76 kB/s | 47 kB 00:00 2024-05-26T11:49:41.780 INFO:teuthology.orchestra.run.smithi192.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2024-05-26T11:49:41.782 INFO:teuthology.orchestra.run.smithi186.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 190 kB/s | 22 kB 00:00 2024-05-26T11:49:41.813 INFO:teuthology.orchestra.run.smithi192.stdout:(69/159): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-05-26T11:49:41.839 INFO:teuthology.orchestra.run.smithi192.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-26T11:49:41.897 INFO:teuthology.orchestra.run.smithi192.stdout:(71/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-05-26T11:49:41.931 INFO:teuthology.orchestra.run.smithi192.stdout:(72/159): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-05-26T11:49:41.948 INFO:teuthology.orchestra.run.smithi192.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-26T11:49:41.957 INFO:teuthology.orchestra.run.smithi186.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.3 MB/s | 863 kB 00:00 2024-05-26T11:49:41.982 INFO:teuthology.orchestra.run.smithi192.stdout:(74/159): python3-colorama-0.4.6-3.el9.noarch.r 1.8 MB/s | 61 kB 00:00 2024-05-26T11:49:41.983 INFO:teuthology.orchestra.run.smithi156.stdout:(64/159): lua-5.4.4-4.el9.x86_64.rpm 240 kB/s | 188 kB 00:00 2024-05-26T11:49:42.034 INFO:teuthology.orchestra.run.smithi156.stdout:(65/159): lttng-ust-2.12.0-6.el9.x86_64.rpm 337 kB/s | 292 kB 00:00 2024-05-26T11:49:42.065 INFO:teuthology.orchestra.run.smithi192.stdout:(75/159): postgresql-server-13.14-1.el9.x86_64. 23 MB/s | 5.8 MB 00:00 2024-05-26T11:49:42.091 INFO:teuthology.orchestra.run.smithi192.stdout:(76/159): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-05-26T11:49:42.116 INFO:teuthology.orchestra.run.smithi192.stdout:(77/159): python3-jinja2-2.11.3-6.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-05-26T11:49:42.133 INFO:teuthology.orchestra.run.smithi192.stdout:(78/159): python3-jsonpatch-1.21-16.el9.noarch. 613 kB/s | 26 kB 00:00 2024-05-26T11:49:42.149 INFO:teuthology.orchestra.run.smithi192.stdout:(79/159): python3-jsonpointer-2.0-4.el9.noarch. 572 kB/s | 19 kB 00:00 2024-05-26T11:49:42.192 INFO:teuthology.orchestra.run.smithi192.stdout:(80/159): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-05-26T11:49:42.207 INFO:teuthology.orchestra.run.smithi186.stdout:(106/159): grpc-data-1.46.7-10.el9.noarch.rpm 78 kB/s | 19 kB 00:00 2024-05-26T11:49:42.217 INFO:teuthology.orchestra.run.smithi192.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 518 kB/s | 35 kB 00:00 2024-05-26T11:49:42.312 INFO:teuthology.orchestra.run.smithi192.stdout:(82/159): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-05-26T11:49:42.316 INFO:teuthology.orchestra.run.smithi186.stdout:(107/159): gperftools-libs-2.9.1-2.el9.x86_64.r 579 kB/s | 309 kB 00:00 2024-05-26T11:49:42.342 INFO:teuthology.orchestra.run.smithi192.stdout:(83/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.0 MB/s | 250 kB 00:00 2024-05-26T11:49:42.368 INFO:teuthology.orchestra.run.smithi192.stdout:(84/159): python3-oauthlib-3.1.1-5.el9.noarch.r 3.9 MB/s | 222 kB 00:00 2024-05-26T11:49:42.376 INFO:teuthology.orchestra.run.smithi156.stdout:(66/159): openblas-0.3.21-2.el9.x86_64.rpm 57 kB/s | 35 kB 00:00 2024-05-26T11:49:42.393 INFO:teuthology.orchestra.run.smithi192.stdout:(85/159): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-26T11:49:42.416 INFO:teuthology.orchestra.run.smithi186.stdout:(108/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 260 kB/s | 26 kB 00:00 2024-05-26T11:49:42.418 INFO:teuthology.orchestra.run.smithi192.stdout:(86/159): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-05-26T11:49:42.444 INFO:teuthology.orchestra.run.smithi192.stdout:(87/159): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-05-26T11:49:42.469 INFO:teuthology.orchestra.run.smithi192.stdout:(88/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-05-26T11:49:42.494 INFO:teuthology.orchestra.run.smithi192.stdout:(89/159): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-05-26T11:49:42.519 INFO:teuthology.orchestra.run.smithi192.stdout:(90/159): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-26T11:49:42.545 INFO:teuthology.orchestra.run.smithi192.stdout:(91/159): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-05-26T11:49:42.566 INFO:teuthology.orchestra.run.smithi186.stdout:(109/159): abseil-cpp-20211102.0-3.el9.x86_64.r 650 kB/s | 548 kB 00:00 2024-05-26T11:49:42.629 INFO:teuthology.orchestra.run.smithi192.stdout:(92/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 5.1 MB 00:00 2024-05-26T11:49:42.654 INFO:teuthology.orchestra.run.smithi192.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-05-26T11:49:42.679 INFO:teuthology.orchestra.run.smithi192.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 883 kB/s | 44 kB 00:00 2024-05-26T11:49:42.692 INFO:teuthology.orchestra.run.smithi186.stdout:(110/159): libunwind-1.6.2-1.el9.x86_64.rpm 539 kB/s | 67 kB 00:00 2024-05-26T11:49:42.704 INFO:teuthology.orchestra.run.smithi192.stdout:(95/159): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-26T11:49:42.729 INFO:teuthology.orchestra.run.smithi192.stdout:(96/159): qatlib-service-24.02.0-1.el9.x86_64.r 717 kB/s | 36 kB 00:00 2024-05-26T11:49:42.755 INFO:teuthology.orchestra.run.smithi192.stdout:(97/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 943 kB/s | 47 kB 00:00 2024-05-26T11:49:42.788 INFO:teuthology.orchestra.run.smithi192.stdout:(98/159): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 305 kB 00:00 2024-05-26T11:49:42.822 INFO:teuthology.orchestra.run.smithi192.stdout:(99/159): uuid-1.6.2-55.el9.x86_64.rpm 862 kB/s | 58 kB 00:00 2024-05-26T11:49:42.848 INFO:teuthology.orchestra.run.smithi192.stdout:(100/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.6 MB/s | 94 kB 00:00 2024-05-26T11:49:42.873 INFO:teuthology.orchestra.run.smithi192.stdout:(101/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-05-26T11:49:42.884 INFO:teuthology.orchestra.run.smithi186.stdout:(111/159): luarocks-3.9.2-5.el9.noarch.rpm 790 kB/s | 151 kB 00:00 2024-05-26T11:49:42.898 INFO:teuthology.orchestra.run.smithi192.stdout:(102/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 894 kB/s | 45 kB 00:00 2024-05-26T11:49:43.107 INFO:teuthology.orchestra.run.smithi192.stdout:(103/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-05-26T11:49:43.132 INFO:teuthology.orchestra.run.smithi192.stdout:(104/159): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-05-26T11:49:43.142 INFO:teuthology.orchestra.run.smithi186.stdout:(112/159): liboath-2.6.7-2.el9.x86_64.rpm 67 kB/s | 49 kB 00:00 2024-05-26T11:49:43.349 INFO:teuthology.orchestra.run.smithi192.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 1.9 MB/s | 863 kB 00:00 2024-05-26T11:49:43.393 INFO:teuthology.orchestra.run.smithi156.stdout:(67/159): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-05-26T11:49:43.460 INFO:teuthology.orchestra.run.smithi156.stdout:(68/159): openblas-openmp-0.3.21-2.el9.x86_64.r 3.3 MB/s | 4.7 MB 00:01 2024-05-26T11:49:43.499 INFO:teuthology.orchestra.run.smithi192.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 842 kB/s | 309 kB 00:00 2024-05-26T11:49:43.525 INFO:teuthology.orchestra.run.smithi192.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 111 kB/s | 19 kB 00:00 2024-05-26T11:49:43.550 INFO:teuthology.orchestra.run.smithi192.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 1.2 MB/s | 548 kB 00:00 2024-05-26T11:49:43.568 INFO:teuthology.orchestra.run.smithi156.stdout:(69/159): postgresql-13.14-1.el9.x86_64.rpm 1.3 MB/s | 1.6 MB 00:01 2024-05-26T11:49:43.625 INFO:teuthology.orchestra.run.smithi192.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 259 kB/s | 26 kB 00:00 2024-05-26T11:49:43.634 INFO:teuthology.orchestra.run.smithi186.stdout:(113/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.1 MB/s | 839 kB 00:00 2024-05-26T11:49:43.650 INFO:teuthology.orchestra.run.smithi192.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-05-26T11:49:43.660 INFO:teuthology.orchestra.run.smithi186.stdout:(114/159): python3-asyncssh-2.13.2-5.el9.noarch 1.0 MB/s | 548 kB 00:00 2024-05-26T11:49:43.727 INFO:teuthology.orchestra.run.smithi186.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-05-26T11:49:43.743 INFO:teuthology.orchestra.run.smithi186.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-05-26T11:49:43.751 INFO:teuthology.orchestra.run.smithi192.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 537 kB/s | 67 kB 00:00 2024-05-26T11:49:43.776 INFO:teuthology.orchestra.run.smithi192.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-05-26T11:49:43.819 INFO:teuthology.orchestra.run.smithi186.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-26T11:49:43.893 INFO:teuthology.orchestra.run.smithi192.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-05-26T11:49:43.927 INFO:teuthology.orchestra.run.smithi186.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 936 kB/s | 172 kB 00:00 2024-05-26T11:49:44.002 INFO:teuthology.orchestra.run.smithi192.stdout:(114/159): libarrow-9.0.0-11.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-05-26T11:49:44.027 INFO:teuthology.orchestra.run.smithi192.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-05-26T11:49:44.077 INFO:teuthology.orchestra.run.smithi192.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 428 kB/s | 32 kB 00:00 2024-05-26T11:49:44.119 INFO:teuthology.orchestra.run.smithi192.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-26T11:49:44.136 INFO:teuthology.orchestra.run.smithi186.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 1.1 MB/s | 358 kB 00:00 2024-05-26T11:49:44.153 INFO:teuthology.orchestra.run.smithi192.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-26T11:49:44.195 INFO:teuthology.orchestra.run.smithi192.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-26T11:49:44.253 INFO:teuthology.orchestra.run.smithi186.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 932 kB/s | 303 kB 00:00 2024-05-26T11:49:44.253 INFO:teuthology.orchestra.run.smithi192.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-05-26T11:49:44.260 INFO:teuthology.orchestra.run.smithi156.stdout:(70/159): postgresql-private-libs-13.14-1.el9.x 167 kB/s | 133 kB 00:00 2024-05-26T11:49:44.304 INFO:teuthology.orchestra.run.smithi192.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 1.9 MB/s | 206 kB 00:00 2024-05-26T11:49:44.344 INFO:teuthology.orchestra.run.smithi186.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 986 kB/s | 206 kB 00:00 2024-05-26T11:49:44.404 INFO:teuthology.orchestra.run.smithi192.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-05-26T11:49:44.427 INFO:teuthology.orchestra.run.smithi156.stdout:(71/159): postgresql-contrib-13.14-1.el9.x86_64 881 kB/s | 911 kB 00:01 2024-05-26T11:49:44.429 INFO:teuthology.orchestra.run.smithi192.stdout:(123/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.2 MB/s | 839 kB 00:00 2024-05-26T11:49:44.497 INFO:teuthology.orchestra.run.smithi192.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-26T11:49:44.562 INFO:teuthology.orchestra.run.smithi186.stdout:(122/159): python3-grpcio-tools-1.46.7-10.el9.x 667 kB/s | 144 kB 00:00 2024-05-26T11:49:44.596 INFO:teuthology.orchestra.run.smithi192.stdout:(125/159): python3-grpcio-1.46.7-10.el9.x86_64. 5.9 MB/s | 2.0 MB 00:00 2024-05-26T11:49:44.622 INFO:teuthology.orchestra.run.smithi192.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 185 kB/s | 23 kB 00:00 2024-05-26T11:49:44.639 INFO:teuthology.orchestra.run.smithi192.stdout:(127/159): python3-jaraco-classes-3.2.1-5.el9.n 84 kB/s | 18 kB 00:00 2024-05-26T11:49:44.654 INFO:teuthology.orchestra.run.smithi186.stdout:(123/159): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-26T11:49:44.681 INFO:teuthology.orchestra.run.smithi192.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 335 kB/s | 20 kB 00:00 2024-05-26T11:49:44.739 INFO:teuthology.orchestra.run.smithi192.stdout:(129/159): python3-jaraco-functools-3.5.0-2.el9 136 kB/s | 19 kB 00:00 2024-05-26T11:49:44.754 INFO:teuthology.orchestra.run.smithi186.stdout:(124/159): python3-jaraco-classes-3.2.1-5.el9.n 177 kB/s | 18 kB 00:00 2024-05-26T11:49:44.756 INFO:teuthology.orchestra.run.smithi192.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 617 kB/s | 46 kB 00:00 2024-05-26T11:49:44.854 INFO:teuthology.orchestra.run.smithi186.stdout:(125/159): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-05-26T11:49:44.856 INFO:teuthology.orchestra.run.smithi192.stdout:(131/159): python3-natsort-7.1.1-5.el9.noarch.r 577 kB/s | 58 kB 00:00 2024-05-26T11:49:44.886 INFO:teuthology.orchestra.run.smithi156.stdout:(72/159): postgresql-server-13.14-1.el9.x86_64. 4.4 MB/s | 5.8 MB 00:01 2024-05-26T11:49:44.890 INFO:teuthology.orchestra.run.smithi192.stdout:(132/159): python3-more-itertools-8.12.0-2.el9. 524 kB/s | 79 kB 00:00 2024-05-26T11:49:44.932 INFO:teuthology.orchestra.run.smithi192.stdout:(133/159): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-05-26T11:49:44.938 INFO:teuthology.orchestra.run.smithi186.stdout:(126/159): python3-jaraco-functools-3.5.0-2.el9 234 kB/s | 19 kB 00:00 2024-05-26T11:49:44.957 INFO:teuthology.orchestra.run.smithi192.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 245 kB/s | 16 kB 00:00 2024-05-26T11:49:44.999 INFO:teuthology.orchestra.run.smithi192.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.3 MB/s | 90 kB 00:00 2024-05-26T11:49:45.021 INFO:teuthology.orchestra.run.smithi186.stdout:(127/159): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-05-26T11:49:45.033 INFO:teuthology.orchestra.run.smithi192.stdout:(136/159): python3-pyperclip-1.8.2-2.el9.noarch 376 kB/s | 28 kB 00:00 2024-05-26T11:49:45.083 INFO:teuthology.orchestra.run.smithi192.stdout:(137/159): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-26T11:49:45.133 INFO:teuthology.orchestra.run.smithi192.stdout:(138/159): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-05-26T11:49:45.175 INFO:teuthology.orchestra.run.smithi192.stdout:(139/159): python3-kubernetes-26.1.0-2.el9.noar 1.9 MB/s | 1.0 MB 00:00 2024-05-26T11:49:45.186 INFO:teuthology.orchestra.run.smithi156.stdout:(73/159): python3-attrs-20.3.0-7.el9.noarch.rpm 115 kB/s | 87 kB 00:00 2024-05-26T11:49:45.201 INFO:teuthology.orchestra.run.smithi192.stdout:(140/159): python3-rsa-4.9-2.el9.noarch.rpm 505 kB/s | 59 kB 00:00 2024-05-26T11:49:45.276 INFO:teuthology.orchestra.run.smithi192.stdout:(141/159): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-05-26T11:49:45.335 INFO:teuthology.orchestra.run.smithi192.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 321 kB/s | 51 kB 00:00 2024-05-26T11:49:45.360 INFO:teuthology.orchestra.run.smithi192.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-05-26T11:49:45.369 INFO:teuthology.orchestra.run.smithi156.stdout:(74/159): protobuf-3.14.0-13.el9.x86_64.rpm 932 kB/s | 1.0 MB 00:01 2024-05-26T11:49:45.385 INFO:teuthology.orchestra.run.smithi192.stdout:(144/159): python3-tempora-5.0.0-2.el9.noarch.r 142 kB/s | 36 kB 00:00 2024-05-26T11:49:45.435 INFO:teuthology.orchestra.run.smithi192.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-05-26T11:49:45.469 INFO:teuthology.orchestra.run.smithi192.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-05-26T11:49:45.586 INFO:teuthology.orchestra.run.smithi192.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 1.7 MB/s | 427 kB 00:00 2024-05-26T11:49:45.711 INFO:teuthology.orchestra.run.smithi192.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 5.7 MB/s | 1.6 MB 00:00 2024-05-26T11:49:45.828 INFO:teuthology.orchestra.run.smithi192.stdout:(149/159): librados2-19.0.0-3867.g9a4c5853.el9. 9.5 MB/s | 3.4 MB 00:00 2024-05-26T11:49:45.903 INFO:teuthology.orchestra.run.smithi156.stdout:(75/159): python3-colorama-0.4.6-3.el9.noarch.r 85 kB/s | 61 kB 00:00 2024-05-26T11:49:45.922 INFO:teuthology.orchestra.run.smithi186.stdout:(128/159): python3-kubernetes-26.1.0-2.el9.noar 1.1 MB/s | 1.0 MB 00:00 2024-05-26T11:49:45.954 INFO:teuthology.orchestra.run.smithi192.stdout:(150/159): NetworkManager-1.47.90-1.el9.x86_64. 9.5 MB/s | 2.3 MB 00:00 2024-05-26T11:49:45.972 INFO:teuthology.orchestra.run.smithi186.stdout:(129/159): python3-grpcio-1.46.7-10.el9.x86_64. 1.2 MB/s | 2.0 MB 00:01 2024-05-26T11:49:46.004 INFO:teuthology.orchestra.run.smithi192.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 838 kB/s | 41 kB 00:00 2024-05-26T11:49:46.014 INFO:teuthology.orchestra.run.smithi186.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 504 kB/s | 46 kB 00:00 2024-05-26T11:49:46.054 INFO:teuthology.orchestra.run.smithi192.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 4.9 MB/s | 248 kB 00:00 2024-05-26T11:49:46.072 INFO:teuthology.orchestra.run.smithi186.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 788 kB/s | 79 kB 00:00 2024-05-26T11:49:46.106 INFO:teuthology.orchestra.run.smithi186.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-05-26T11:49:46.113 INFO:teuthology.orchestra.run.smithi192.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 6.4 MB/s | 1.8 MB 00:00 2024-05-26T11:49:46.138 INFO:teuthology.orchestra.run.smithi192.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 439 kB/s | 37 kB 00:00 2024-05-26T11:49:46.162 INFO:teuthology.orchestra.run.smithi156.stdout:(76/159): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-05-26T11:49:46.163 INFO:teuthology.orchestra.run.smithi192.stdout:(155/159): elfutils-libelf-0.191-4.el9.x86_64.r 4.0 MB/s | 207 kB 00:00 2024-05-26T11:49:46.191 INFO:teuthology.orchestra.run.smithi192.stdout:(156/159): elfutils-libs-0.191-4.el9.x86_64.rpm 4.9 MB/s | 260 kB 00:00 2024-05-26T11:49:46.206 INFO:teuthology.orchestra.run.smithi186.stdout:(133/159): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-05-26T11:49:46.212 INFO:teuthology.orchestra.run.smithi156.stdout:(77/159): python3-devel-3.9.18-3.el9.x86_64.rpm 290 kB/s | 244 kB 00:00 2024-05-26T11:49:46.241 INFO:teuthology.orchestra.run.smithi192.stdout:(157/159): openssl-3.2.1-1.el9.x86_64.rpm 17 MB/s | 1.3 MB 00:00 2024-05-26T11:49:46.265 INFO:teuthology.orchestra.run.smithi186.stdout:(134/159): libarrow-9.0.0-11.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:04 2024-05-26T11:49:46.300 INFO:teuthology.orchestra.run.smithi192.stdout:(158/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2024-05-26T11:49:46.307 INFO:teuthology.orchestra.run.smithi186.stdout:(135/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 897 kB/s | 90 kB 00:00 2024-05-26T11:49:46.349 INFO:teuthology.orchestra.run.smithi186.stdout:(136/159): python3-pecan-1.4.2-3.el9.noarch.rpm 986 kB/s | 272 kB 00:00 2024-05-26T11:49:46.366 INFO:teuthology.orchestra.run.smithi186.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-05-26T11:49:46.391 INFO:teuthology.orchestra.run.smithi186.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 366 kB/s | 31 kB 00:00 2024-05-26T11:49:46.466 INFO:teuthology.orchestra.run.smithi186.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 590 kB/s | 59 kB 00:00 2024-05-26T11:49:46.492 INFO:teuthology.orchestra.run.smithi186.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-05-26T11:49:46.517 INFO:teuthology.orchestra.run.smithi186.stdout:(141/159): python3-routes-2.5.1-5.el9.noarch.rp 1.1 MB/s | 188 kB 00:00 2024-05-26T11:49:46.567 INFO:teuthology.orchestra.run.smithi186.stdout:(142/159): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-05-26T11:49:46.608 INFO:teuthology.orchestra.run.smithi192.stdout:(159/159): librbd1-19.0.0-3867.g9a4c5853.el9.x8 3.1 MB/s | 3.1 MB 00:01 2024-05-26T11:49:46.611 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-05-26T11:49:46.611 INFO:teuthology.orchestra.run.smithi192.stdout:Total 12 MB/s | 227 MB 00:19 2024-05-26T11:49:46.617 INFO:teuthology.orchestra.run.smithi186.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-05-26T11:49:46.668 INFO:teuthology.orchestra.run.smithi186.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 1.3 MB/s | 230 kB 00:00 2024-05-26T11:49:46.771 INFO:teuthology.orchestra.run.smithi156.stdout:(78/159): python3-jinja2-2.11.3-6.el9.noarch.rp 286 kB/s | 248 kB 00:00 2024-05-26T11:49:46.796 INFO:teuthology.orchestra.run.smithi156.stdout:(79/159): python3-jsonpatch-1.21-16.el9.noarch. 42 kB/s | 26 kB 00:00 2024-05-26T11:49:46.821 INFO:teuthology.orchestra.run.smithi156.stdout:(80/159): python3-jsonpointer-2.0-4.el9.noarch. 31 kB/s | 19 kB 00:00 2024-05-26T11:49:46.904 INFO:teuthology.orchestra.run.smithi186.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 70 kB/s | 20 kB 00:00 2024-05-26T11:49:46.935 INFO:teuthology.orchestra.run.smithi186.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 718 kB/s | 192 kB 00:00 2024-05-26T11:49:46.961 INFO:teuthology.orchestra.run.smithi186.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 1.1 MB/s | 427 kB 00:00 2024-05-26T11:49:47.261 INFO:teuthology.orchestra.run.smithi186.stdout:(148/159): librbd1-19.0.0-3867.g9a4c5853.el9.x8 10 MB/s | 3.1 MB 00:00 2024-05-26T11:49:47.446 INFO:teuthology.orchestra.run.smithi156.stdout:(81/159): python3-markupsafe-1.1.1-12.el9.x86_6 56 kB/s | 35 kB 00:00 2024-05-26T11:49:47.461 INFO:teuthology.orchestra.run.smithi186.stdout:(149/159): NetworkManager-1.47.90-1.el9.x86_64. 12 MB/s | 2.3 MB 00:00 2024-05-26T11:49:47.528 INFO:teuthology.orchestra.run.smithi186.stdout:(150/159): NetworkManager-libnm-1.47.90-1.el9.x 27 MB/s | 1.8 MB 00:00 2024-05-26T11:49:47.570 INFO:teuthology.orchestra.run.smithi186.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 1.0 MB/s | 41 kB 00:00 2024-05-26T11:49:47.612 INFO:teuthology.orchestra.run.smithi186.stdout:(152/159): NetworkManager-tui-1.47.90-1.el9.x86 5.8 MB/s | 248 kB 00:00 2024-05-26T11:49:47.613 INFO:teuthology.orchestra.run.smithi156.stdout:(82/159): python3-mako-1.1.4-6.el9.noarch.rpm 205 kB/s | 172 kB 00:00 2024-05-26T11:49:47.654 INFO:teuthology.orchestra.run.smithi186.stdout:(153/159): elfutils-debuginfod-client-0.191-4.e 885 kB/s | 37 kB 00:00 2024-05-26T11:49:47.696 INFO:teuthology.orchestra.run.smithi186.stdout:(154/159): elfutils-libelf-0.191-4.el9.x86_64.r 4.9 MB/s | 207 kB 00:00 2024-05-26T11:49:47.746 INFO:teuthology.orchestra.run.smithi186.stdout:(155/159): elfutils-libs-0.191-4.el9.x86_64.rpm 5.1 MB/s | 260 kB 00:00 2024-05-26T11:49:47.813 INFO:teuthology.orchestra.run.smithi186.stdout:(156/159): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-05-26T11:49:47.833 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-05-26T11:49:47.888 INFO:teuthology.orchestra.run.smithi186.stdout:(157/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 32 MB/s | 2.4 MB 00:00 2024-05-26T11:49:47.988 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-05-26T11:49:47.988 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-05-26T11:49:48.130 INFO:teuthology.orchestra.run.smithi186.stdout:(158/159): librados2-19.0.0-3867.g9a4c5853.el9. 2.8 MB/s | 3.4 MB 00:01 2024-05-26T11:49:48.138 INFO:teuthology.orchestra.run.smithi156.stdout:(83/159): python3-numpy-1.20.1-5.el9.x86_64.rpm 3.9 MB/s | 5.1 MB 00:01 2024-05-26T11:49:48.180 INFO:teuthology.orchestra.run.smithi186.stdout:(159/159): thrift-0.15.0-2.el9.x86_64.rpm 1.2 MB/s | 1.6 MB 00:01 2024-05-26T11:49:48.183 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2024-05-26T11:49:48.183 INFO:teuthology.orchestra.run.smithi186.stdout:Total 10 MB/s | 227 MB 00:21 2024-05-26T11:49:48.305 INFO:teuthology.orchestra.run.smithi156.stdout:(84/159): python3-numpy-f2py-1.20.1-5.el9.x86_6 292 kB/s | 250 kB 00:00 2024-05-26T11:49:48.456 INFO:teuthology.orchestra.run.smithi156.stdout:(85/159): python3-oauthlib-3.1.1-5.el9.noarch.r 264 kB/s | 222 kB 00:00 2024-05-26T11:49:48.856 INFO:teuthology.orchestra.run.smithi156.stdout:(86/159): python3-packaging-20.9-5.el9.noarch.r 108 kB/s | 77 kB 00:00 2024-05-26T11:49:48.940 INFO:teuthology.orchestra.run.smithi156.stdout:(87/159): python3-prettytable-0.7.2-27.el9.noar 67 kB/s | 42 kB 00:00 2024-05-26T11:49:49.298 INFO:teuthology.orchestra.run.smithi156.stdout:(88/159): python3-protobuf-3.14.0-13.el9.noarch 320 kB/s | 269 kB 00:00 2024-05-26T11:49:49.364 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2024-05-26T11:49:49.527 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2024-05-26T11:49:49.528 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2024-05-26T11:49:49.623 INFO:teuthology.orchestra.run.smithi156.stdout:(89/159): python3-pyasn1-0.4.8-6.el9.noarch.rpm 207 kB/s | 159 kB 00:00 2024-05-26T11:49:49.757 INFO:teuthology.orchestra.run.smithi156.stdout:(90/159): python3-pyasn1-modules-0.4.8-6.el9.no 341 kB/s | 279 kB 00:00 2024-05-26T11:49:49.791 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-05-26T11:49:49.792 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-05-26T11:49:49.949 INFO:teuthology.orchestra.run.smithi156.stdout:(91/159): python3-pytz-2021.1-5.el9.noarch.rpm 78 kB/s | 51 kB 00:00 2024-05-26T11:49:50.274 INFO:teuthology.orchestra.run.smithi156.stdout:(92/159): python3-requests-oauthlib-1.3.0-12.el 83 kB/s | 54 kB 00:00 2024-05-26T11:49:50.566 INFO:teuthology.orchestra.run.smithi156.stdout:(93/159): python3-toml-0.10.2-6.el9.noarch.rpm 68 kB/s | 42 kB 00:00 2024-05-26T11:49:50.916 INFO:teuthology.orchestra.run.smithi156.stdout:(94/159): python3-wcwidth-0.2.5-8.el9.noarch.rp 69 kB/s | 44 kB 00:00 2024-05-26T11:49:51.334 INFO:teuthology.orchestra.run.smithi156.stdout:(95/159): python3-scipy-1.6.2-8.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-05-26T11:49:51.434 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2024-05-26T11:49:51.435 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2024-05-26T11:49:51.450 INFO:teuthology.orchestra.run.smithi156.stdout:(96/159): qatlib-24.02.0-1.el9.x86_64.rpm 250 kB/s | 221 kB 00:00 2024-05-26T11:49:51.617 INFO:teuthology.orchestra.run.smithi156.stdout:(97/159): qatlib-service-24.02.0-1.el9.x86_64.r 51 kB/s | 36 kB 00:00 2024-05-26T11:49:51.951 INFO:teuthology.orchestra.run.smithi156.stdout:(98/159): qatzip-libs-1.2.0-1.el9.x86_64.rpm 77 kB/s | 47 kB 00:00 2024-05-26T11:49:52.276 INFO:teuthology.orchestra.run.smithi156.stdout:(99/159): socat-1.7.4.1-5.el9.x86_64.rpm 369 kB/s | 305 kB 00:00 2024-05-26T11:49:52.301 INFO:teuthology.orchestra.run.smithi156.stdout:(100/159): uuid-1.6.2-55.el9.x86_64.rpm 84 kB/s | 58 kB 00:00 2024-05-26T11:49:52.660 INFO:teuthology.orchestra.run.smithi156.stdout:(101/159): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 132 kB/s | 94 kB 00:00 2024-05-26T11:49:52.752 INFO:teuthology.orchestra.run.smithi156.stdout:(102/159): lua-devel-5.4.4-4.el9.x86_64.rpm 244 kB/s | 22 kB 00:00 2024-05-26T11:49:52.927 INFO:teuthology.orchestra.run.smithi156.stdout:(103/159): zlib-devel-1.2.11-41.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-05-26T11:49:52.952 INFO:teuthology.orchestra.run.smithi156.stdout:(104/159): xmlstarlet-1.6.1-20.el9.x86_64.rpm 94 kB/s | 64 kB 00:00 2024-05-26T11:49:53.003 INFO:teuthology.orchestra.run.smithi156.stdout:(105/159): protobuf-compiler-3.14.0-13.el9.x86_ 3.4 MB/s | 863 kB 00:00 2024-05-26T11:49:53.053 INFO:teuthology.orchestra.run.smithi156.stdout:(106/159): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-05-26T11:49:53.078 INFO:teuthology.orchestra.run.smithi156.stdout:(107/159): grpc-data-1.46.7-10.el9.noarch.rpm 258 kB/s | 19 kB 00:00 2024-05-26T11:49:53.103 INFO:teuthology.orchestra.run.smithi156.stdout:(108/159): abseil-cpp-20211102.0-3.el9.x86_64.r 3.1 MB/s | 548 kB 00:00 2024-05-26T11:49:53.129 INFO:teuthology.orchestra.run.smithi156.stdout:(109/159): libarrow-doc-9.0.0-11.el9.noarch.rpm 518 kB/s | 26 kB 00:00 2024-05-26T11:49:53.154 INFO:teuthology.orchestra.run.smithi156.stdout:(110/159): liboath-2.6.7-2.el9.x86_64.rpm 966 kB/s | 49 kB 00:00 2024-05-26T11:49:53.179 INFO:teuthology.orchestra.run.smithi156.stdout:(111/159): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-05-26T11:49:53.204 INFO:teuthology.orchestra.run.smithi156.stdout:(112/159): luarocks-3.9.2-5.el9.noarch.rpm 3.0 MB/s | 151 kB 00:00 2024-05-26T11:49:53.263 INFO:teuthology.orchestra.run.smithi156.stdout:(113/159): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-05-26T11:49:53.289 INFO:teuthology.orchestra.run.smithi156.stdout:(114/159): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.5 MB/s | 839 kB 00:00 2024-05-26T11:49:53.314 INFO:teuthology.orchestra.run.smithi156.stdout:(115/159): python3-bcrypt-3.2.2-1.el9.x86_64.rp 865 kB/s | 43 kB 00:00 2024-05-26T11:49:53.339 INFO:teuthology.orchestra.run.smithi156.stdout:(116/159): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-05-26T11:49:53.364 INFO:teuthology.orchestra.run.smithi156.stdout:(117/159): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-05-26T11:49:53.390 INFO:teuthology.orchestra.run.smithi156.stdout:(118/159): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-05-26T11:49:53.424 INFO:teuthology.orchestra.run.smithi156.stdout:(119/159): python3-cherrypy-18.6.1-2.el9.noarch 6.0 MB/s | 358 kB 00:00 2024-05-26T11:49:53.449 INFO:teuthology.orchestra.run.smithi156.stdout:(120/159): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.0 MB/s | 303 kB 00:00 2024-05-26T11:49:53.474 INFO:teuthology.orchestra.run.smithi156.stdout:(121/159): python3-google-auth-2.29.0-1.el9.noa 4.0 MB/s | 206 kB 00:00 2024-05-26T11:49:53.541 INFO:teuthology.orchestra.run.smithi156.stdout:(122/159): libarrow-9.0.0-11.el9.x86_64.rpm 9.0 MB/s | 4.4 MB 00:00 2024-05-26T11:49:53.586 INFO:teuthology.orchestra.run.smithi156.stdout:(123/159): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-05-26T11:49:53.617 INFO:teuthology.orchestra.run.smithi156.stdout:(124/159): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-05-26T11:49:53.642 INFO:teuthology.orchestra.run.smithi156.stdout:(125/159): python3-jaraco-classes-3.2.1-5.el9.n 319 kB/s | 18 kB 00:00 2024-05-26T11:49:53.667 INFO:teuthology.orchestra.run.smithi156.stdout:(126/159): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-05-26T11:49:53.693 INFO:teuthology.orchestra.run.smithi156.stdout:(127/159): python3-jaraco-functools-3.5.0-2.el9 386 kB/s | 19 kB 00:00 2024-05-26T11:49:53.718 INFO:teuthology.orchestra.run.smithi156.stdout:(128/159): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-05-26T11:49:53.760 INFO:teuthology.orchestra.run.smithi156.stdout:(129/159): python3-grpcio-1.46.7-10.el9.x86_64. 6.6 MB/s | 2.0 MB 00:00 2024-05-26T11:49:53.785 INFO:teuthology.orchestra.run.smithi156.stdout:(130/159): python3-logutils-0.3.5-21.el9.noarch 693 kB/s | 46 kB 00:00 2024-05-26T11:49:53.810 INFO:teuthology.orchestra.run.smithi156.stdout:(131/159): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-26T11:49:53.836 INFO:teuthology.orchestra.run.smithi156.stdout:(132/159): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-26T11:49:53.869 INFO:teuthology.orchestra.run.smithi156.stdout:(133/159): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2024-05-26T11:49:53.895 INFO:teuthology.orchestra.run.smithi156.stdout:(134/159): python3-portend-3.1.0-2.el9.noarch.r 280 kB/s | 16 kB 00:00 2024-05-26T11:49:53.920 INFO:teuthology.orchestra.run.smithi156.stdout:(135/159): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-05-26T11:49:53.945 INFO:teuthology.orchestra.run.smithi156.stdout:(136/159): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-26T11:49:53.971 INFO:teuthology.orchestra.run.smithi156.stdout:(137/159): python3-pyperclip-1.8.2-2.el9.noarch 374 kB/s | 28 kB 00:00 2024-05-26T11:49:53.996 INFO:teuthology.orchestra.run.smithi156.stdout:(138/159): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-05-26T11:49:54.021 INFO:teuthology.orchestra.run.smithi156.stdout:(139/159): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-05-26T11:49:54.046 INFO:teuthology.orchestra.run.smithi156.stdout:(140/159): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-05-26T11:49:54.072 INFO:teuthology.orchestra.run.smithi156.stdout:(141/159): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-26T11:49:54.097 INFO:teuthology.orchestra.run.smithi156.stdout:(142/159): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-05-26T11:49:54.122 INFO:teuthology.orchestra.run.smithi156.stdout:(143/159): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-05-26T11:49:54.148 INFO:teuthology.orchestra.run.smithi156.stdout:(144/159): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-26T11:49:54.173 INFO:teuthology.orchestra.run.smithi156.stdout:(145/159): python3-zc-lockfile-2.0-10.el9.noarc 396 kB/s | 20 kB 00:00 2024-05-26T11:49:54.198 INFO:teuthology.orchestra.run.smithi156.stdout:(146/159): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-26T11:49:54.224 INFO:teuthology.orchestra.run.smithi156.stdout:(147/159): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-26T11:49:54.291 INFO:teuthology.orchestra.run.smithi156.stdout:(148/159): thrift-0.15.0-2.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-05-26T11:49:54.566 INFO:teuthology.orchestra.run.smithi156.stdout:(149/159): librados2-19.0.0-3867.g9a4c5853.el9. 9.2 MB/s | 3.4 MB 00:00 2024-05-26T11:49:54.751 INFO:teuthology.orchestra.run.smithi156.stdout:(150/159): librbd1-19.0.0-3867.g9a4c5853.el9.x8 6.0 MB/s | 3.1 MB 00:00 2024-05-26T11:49:55.358 INFO:teuthology.orchestra.run.smithi156.stdout:(151/159): NetworkManager-team-1.47.90-1.el9.x8 68 kB/s | 41 kB 00:00 2024-05-26T11:49:55.417 INFO:teuthology.orchestra.run.smithi156.stdout:(152/159): NetworkManager-1.47.90-1.el9.x86_64. 2.1 MB/s | 2.3 MB 00:01 2024-05-26T11:49:55.709 INFO:teuthology.orchestra.run.smithi156.stdout:(153/159): NetworkManager-libnm-1.47.90-1.el9.x 1.6 MB/s | 1.8 MB 00:01 2024-05-26T11:49:56.017 INFO:teuthology.orchestra.run.smithi156.stdout:(154/159): elfutils-debuginfod-client-0.191-4.e 61 kB/s | 37 kB 00:00 2024-05-26T11:49:56.076 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-05-26T11:49:56.138 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-26T11:49:56.176 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-26T11:49:56.201 INFO:teuthology.orchestra.run.smithi156.stdout:(155/159): NetworkManager-tui-1.47.90-1.el9.x86 294 kB/s | 248 kB 00:00 2024-05-26T11:49:56.230 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-26T11:49:56.263 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-26T11:49:56.289 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-26T11:49:56.332 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-26T11:49:56.551 INFO:teuthology.orchestra.run.smithi156.stdout:(156/159): elfutils-libelf-0.191-4.el9.x86_64.r 246 kB/s | 207 kB 00:00 2024-05-26T11:49:56.670 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-26T11:49:56.774 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:49:56.879 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:49:56.917 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:49:56.926 INFO:teuthology.orchestra.run.smithi156.stdout:(157/159): elfutils-libs-0.191-4.el9.x86_64.rpm 286 kB/s | 260 kB 00:00 2024-05-26T11:49:56.992 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:49:57.021 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:49:57.054 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-26T11:49:57.240 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-26T11:49:57.295 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-26T11:49:57.323 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:49:57.410 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:49:57.432 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:49:57.478 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:49:57.500 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:49:57.518 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:49:57.552 INFO:teuthology.orchestra.run.smithi156.stdout:(158/159): openssl-3.2.1-1.el9.x86_64.rpm 1.0 MB/s | 1.3 MB 00:01 2024-05-26T11:49:57.686 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2024-05-26T11:49:57.700 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:49:57.776 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-26T11:49:57.814 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-26T11:49:57.827 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:49:57.868 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-26T11:49:57.910 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-26T11:49:57.936 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-26T11:49:57.970 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-26T11:49:57.985 INFO:teuthology.orchestra.run.smithi156.stdout:(159/159): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.7 MB/s | 2.4 MB 00:01 2024-05-26T11:49:57.989 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-05-26T11:49:57.990 INFO:teuthology.orchestra.run.smithi156.stdout:Total 6.9 MB/s | 227 MB 00:33 2024-05-26T11:49:58.006 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:49:58.066 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:49:58.163 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:49:58.251 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:49:58.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-26T11:49:58.403 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:49:58.497 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:49:58.525 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:49:58.563 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:49:58.612 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-26T11:49:58.640 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-26T11:49:58.654 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:49:58.681 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-26T11:49:58.683 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:49:58.716 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-26T11:49:58.907 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-26T11:49:58.957 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-26T11:49:58.984 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:49:59.017 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-26T11:49:59.071 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:49:59.094 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:49:59.140 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:49:59.161 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:49:59.180 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:49:59.199 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-26T11:49:59.241 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction check 2024-05-26T11:49:59.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-26T11:49:59.284 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-26T11:49:59.312 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-26T11:49:59.362 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:49:59.381 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-26T11:49:59.399 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction check succeeded. 2024-05-26T11:49:59.399 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction test 2024-05-26T11:49:59.434 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-26T11:49:59.464 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-26T11:49:59.491 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:49:59.500 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-26T11:49:59.577 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-26T11:49:59.619 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 33/170 2024-05-26T11:49:59.658 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:49:59.663 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:49:59.692 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 35/170 2024-05-26T11:49:59.710 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:49:59.763 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-26T11:49:59.807 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-26T11:49:59.810 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:49:59.882 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-26T11:49:59.904 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:50:00.014 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-26T11:50:00.068 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-26T11:50:00.122 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:50:00.231 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-26T11:50:00.258 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-26T11:50:00.293 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-26T11:50:00.358 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-26T11:50:00.392 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-26T11:50:00.436 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-26T11:50:00.494 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-26T11:50:00.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-26T11:50:00.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-26T11:50:00.590 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-26T11:50:00.643 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-26T11:50:00.672 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-26T11:50:00.674 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-26T11:50:00.701 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-26T11:50:00.709 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-26T11:50:00.737 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-26T11:50:00.786 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-26T11:50:00.786 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-26T11:50:00.820 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 33/170 2024-05-26T11:50:00.824 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 46/170 2024-05-26T11:50:00.864 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:50:00.866 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-26T11:50:00.892 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 35/170 2024-05-26T11:50:00.963 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-26T11:50:01.007 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-26T11:50:01.007 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-26T11:50:01.061 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-26T11:50:01.081 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-26T11:50:01.152 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-26T11:50:01.214 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-26T11:50:01.217 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-26T11:50:01.262 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-26T11:50:01.281 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-26T11:50:01.312 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-26T11:50:01.349 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-26T11:50:01.395 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-26T11:50:01.440 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-26T11:50:01.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-26T11:50:01.526 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-26T11:50:01.663 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-26T11:50:01.682 INFO:teuthology.orchestra.run.smithi156.stdout:Transaction test succeeded. 2024-05-26T11:50:01.682 INFO:teuthology.orchestra.run.smithi156.stdout:Running transaction 2024-05-26T11:50:01.702 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-26T11:50:01.744 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-26T11:50:01.790 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-26T11:50:01.865 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-26T11:50:01.894 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-26T11:50:01.930 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-26T11:50:01.971 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-26T11:50:02.009 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 46/170 2024-05-26T11:50:02.044 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-26T11:50:02.058 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-26T11:50:02.078 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-26T11:50:02.118 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-26T11:50:02.144 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-26T11:50:02.168 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-26T11:50:02.206 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-26T11:50:02.244 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-26T11:50:02.262 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-26T11:50:02.354 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-26T11:50:02.370 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-26T11:50:02.409 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-26T11:50:02.417 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-26T11:50:02.437 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-26T11:50:02.471 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-26T11:50:02.490 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-26T11:50:02.510 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-26T11:50:02.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-26T11:50:02.550 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-26T11:50:02.550 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-26T11:50:02.581 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-26T11:50:02.597 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-26T11:50:02.615 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-26T11:50:02.632 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-26T11:50:02.648 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-26T11:50:02.675 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-26T11:50:02.690 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-26T11:50:02.717 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-26T11:50:02.724 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-26T11:50:02.811 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-26T11:50:02.853 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-26T11:50:02.863 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-26T11:50:02.886 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-26T11:50:02.897 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-26T11:50:02.927 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-26T11:50:02.942 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-26T11:50:03.020 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-26T11:50:03.040 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-26T11:50:03.061 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-26T11:50:03.088 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-26T11:50:03.102 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-26T11:50:03.127 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-26T11:50:03.151 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-26T11:50:03.225 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-26T11:50:03.352 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-26T11:50:03.392 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-26T11:50:03.420 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-26T11:50:03.455 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-26T11:50:03.486 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-26T11:50:03.525 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-26T11:50:03.563 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-26T11:50:03.598 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-26T11:50:03.631 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-26T11:50:03.673 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-26T11:50:03.707 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-26T11:50:03.795 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-26T11:50:03.812 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-26T11:50:03.837 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-26T11:50:03.860 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-26T11:50:03.869 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-26T11:50:03.894 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-26T11:50:03.907 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-26T11:50:03.993 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-26T11:50:04.005 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-26T11:50:04.041 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-26T11:50:04.054 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-26T11:50:04.087 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-26T11:50:04.174 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-26T11:50:04.222 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-26T11:50:04.402 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-26T11:50:04.428 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-26T11:50:04.507 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-26T11:50:04.773 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-26T11:50:04.818 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-26T11:50:04.852 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-26T11:50:04.950 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-26T11:50:05.006 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-26T11:50:05.052 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-26T11:50:05.131 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-26T11:50:05.149 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-26T11:50:05.171 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-26T11:50:05.261 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-26T11:50:05.349 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-26T11:50:05.377 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-26T11:50:05.455 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-26T11:50:06.099 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-26T11:50:06.208 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-26T11:50:06.484 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-26T11:50:06.576 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-26T11:50:06.604 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-26T11:50:06.637 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-26T11:50:06.949 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-26T11:50:06.992 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:07.066 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:07.096 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 104/170 2024-05-26T11:50:07.132 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-26T11:50:07.431 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-26T11:50:07.498 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-26T11:50:07.526 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-26T11:50:07.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-26T11:50:07.613 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-26T11:50:07.645 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:07.716 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:07.738 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 108/170 2024-05-26T11:50:07.854 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-26T11:50:07.898 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:07.932 INFO:teuthology.orchestra.run.smithi156.stdout: Preparing : 1/1 2024-05-26T11:50:07.987 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:08.010 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 104/170 2024-05-26T11:50:08.031 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/170 2024-05-26T11:50:08.046 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-26T11:50:08.070 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/170 2024-05-26T11:50:08.124 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/170 2024-05-26T11:50:08.156 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/170 2024-05-26T11:50:08.191 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/170 2024-05-26T11:50:08.237 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/170 2024-05-26T11:50:08.525 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-26T11:50:08.559 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:08.614 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/170 2024-05-26T11:50:08.638 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:08.660 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 108/170 2024-05-26T11:50:08.675 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:50:08.798 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:50:08.826 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:50:08.918 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:50:08.947 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:50:09.003 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/170 2024-05-26T11:50:09.220 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 12/170 2024-05-26T11:50:09.271 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-13.14-1.el9.x86_64 13/170 2024-05-26T11:50:09.298 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:50:09.410 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:50:09.433 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:50:09.504 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 15/170 2024-05-26T11:50:09.525 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:50:09.548 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 16/170 2024-05-26T11:50:09.720 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:50:09.919 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 17/170 2024-05-26T11:50:09.974 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:10.010 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:10.056 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:50:10.116 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 18/170 2024-05-26T11:50:10.222 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:50:10.301 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:50:10.506 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.90-1.el9.x86_64 19/170 2024-05-26T11:50:10.549 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:10.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:10.616 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : re2-1:20211101-3.el9.x86_64 20/170 2024-05-26T11:50:10.647 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 21/170 2024-05-26T11:50:10.671 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:10.671 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-26T11:50:10.671 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-26T11:50:10.671 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:10.681 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/170 2024-05-26T11:50:10.698 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:10.754 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/170 2024-05-26T11:50:10.789 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/170 2024-05-26T11:50:10.832 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/170 2024-05-26T11:50:10.866 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/170 2024-05-26T11:50:10.893 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/170 2024-05-26T11:50:10.949 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:10.962 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/170 2024-05-26T11:50:10.982 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:11.015 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/170 2024-05-26T11:50:11.047 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/170 2024-05-26T11:50:11.082 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/170 2024-05-26T11:50:11.158 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/170 2024-05-26T11:50:11.193 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 33/170 2024-05-26T11:50:11.231 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:50:11.256 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 35/170 2024-05-26T11:50:11.325 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/170 2024-05-26T11:50:11.371 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : luarocks-3.9.2-5.el9.noarch 37/170 2024-05-26T11:50:11.446 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/170 2024-05-26T11:50:11.520 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:11.560 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:11.578 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/170 2024-05-26T11:50:11.632 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/170 2024-05-26T11:50:11.651 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:11.652 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-26T11:50:11.652 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-26T11:50:11.652 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:11.678 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:12.152 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/170 2024-05-26T11:50:12.238 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/170 2024-05-26T11:50:12.273 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 43/170 2024-05-26T11:50:12.312 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/170 2024-05-26T11:50:12.359 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 45/170 2024-05-26T11:50:12.397 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 46/170 2024-05-26T11:50:12.439 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 47/170 2024-05-26T11:50:12.593 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 48/170 2024-05-26T11:50:12.642 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 49/170 2024-05-26T11:50:12.732 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 50/170 2024-05-26T11:50:12.797 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 51/170 2024-05-26T11:50:12.870 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 52/170 2024-05-26T11:50:12.910 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 53/170 2024-05-26T11:50:12.946 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 54/170 2024-05-26T11:50:12.993 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 55/170 2024-05-26T11:50:13.029 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 56/170 2024-05-26T11:50:13.072 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 57/170 2024-05-26T11:50:13.114 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 58/170 2024-05-26T11:50:13.251 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 59/170 2024-05-26T11:50:13.291 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 60/170 2024-05-26T11:50:13.333 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 61/170 2024-05-26T11:50:13.428 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 62/170 2024-05-26T11:50:13.458 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 63/170 2024-05-26T11:50:13.499 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 64/170 2024-05-26T11:50:13.532 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 65/170 2024-05-26T11:50:13.556 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 66/170 2024-05-26T11:50:13.643 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 67/170 2024-05-26T11:50:13.769 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 68/170 2024-05-26T11:50:13.816 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 69/170 2024-05-26T11:50:13.853 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 70/170 2024-05-26T11:50:13.887 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 71/170 2024-05-26T11:50:13.925 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : uuid-1.6.2-55.el9.x86_64 72/170 2024-05-26T11:50:13.957 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 73/170 2024-05-26T11:50:13.987 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 74/170 2024-05-26T11:50:14.022 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 75/170 2024-05-26T11:50:14.055 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 76/170 2024-05-26T11:50:14.096 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 77/170 2024-05-26T11:50:14.130 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 78/170 2024-05-26T11:50:14.217 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 79/170 2024-05-26T11:50:14.271 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 80/170 2024-05-26T11:50:14.301 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 81/170 2024-05-26T11:50:14.347 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 82/170 2024-05-26T11:50:14.446 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-packaging-20.9-5.el9.noarch 83/170 2024-05-26T11:50:14.494 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 84/170 2024-05-26T11:50:15.220 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 85/170 2024-05-26T11:50:15.266 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 86/170 2024-05-26T11:50:15.299 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 87/170 2024-05-26T11:50:15.398 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 88/170 2024-05-26T11:50:15.454 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 89/170 2024-05-26T11:50:15.509 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 90/170 2024-05-26T11:50:15.595 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 91/170 2024-05-26T11:50:15.627 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 92/170 2024-05-26T11:50:15.806 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-0.3.21-2.el9.x86_64 93/170 2024-05-26T11:50:15.833 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 94/170 2024-05-26T11:50:15.912 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 95/170 2024-05-26T11:50:16.561 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 96/170 2024-05-26T11:50:16.691 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 97/170 2024-05-26T11:50:17.959 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 98/170 2024-05-26T11:50:18.053 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 99/170 2024-05-26T11:50:18.081 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 100/170 2024-05-26T11:50:18.114 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 101/170 2024-05-26T11:50:18.434 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 102/170 2024-05-26T11:50:18.485 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:18.576 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 103/170 2024-05-26T11:50:18.607 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 104/170 2024-05-26T11:50:18.652 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 105/170 2024-05-26T11:50:19.131 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 106/170 2024-05-26T11:50:19.155 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:19.218 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 107/170 2024-05-26T11:50:19.248 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 108/170 2024-05-26T11:50:21.515 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:21.544 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:22.113 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 109/170 2024-05-26T11:50:22.156 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:22.232 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 110/170 2024-05-26T11:50:22.232 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-26T11:50:22.232 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-26T11:50:22.232 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:22.265 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:22.469 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-05-26T11:50:22.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:22.657 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:22.673 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:22.673 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:22.673 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-26T11:50:22.673 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:23.282 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2024-05-26T11:50:23.283 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:23.450 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:23.465 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:23.465 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:23.465 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-26T11:50:23.465 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:24.160 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:24.187 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:24.434 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:24.878 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:24.901 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:24.901 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:24.902 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-26T11:50:24.902 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:24.902 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:24.902 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:24.936 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:24.963 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:25.008 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-26T11:50:25.065 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-26T11:50:25.120 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-26T11:50:25.187 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:25.210 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:25.308 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-26T11:50:25.363 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-26T11:50:25.441 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-26T11:50:25.529 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-26T11:50:25.646 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:25.668 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:25.669 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-26T11:50:25.712 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-26T11:50:25.745 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-26T11:50:25.772 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-26T11:50:25.776 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-26T11:50:25.833 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-26T11:50:25.889 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-26T11:50:26.075 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-26T11:50:26.122 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-26T11:50:26.140 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:26.170 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:26.200 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-26T11:50:26.203 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 128/170 2024-05-26T11:50:26.264 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 129/170 2024-05-26T11:50:26.293 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:26.297 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-26T11:50:26.437 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-26T11:50:26.489 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-26T11:50:26.521 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-26T11:50:26.548 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-26T11:50:26.934 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:26.956 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:26.979 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 128/170 2024-05-26T11:50:27.032 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 129/170 2024-05-26T11:50:27.062 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:27.342 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:27.378 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:28.110 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:28.147 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:28.346 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:28.370 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:28.482 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:28.650 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 133/170 2024-05-26T11:50:28.689 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:28.713 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:28.765 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:28.776 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:28.875 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 136/170 2024-05-26T11:50:28.921 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 137/170 2024-05-26T11:50:29.115 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:29.139 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:29.250 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:29.409 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 133/170 2024-05-26T11:50:29.441 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:29.465 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:29.509 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:29.519 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:29.617 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 136/170 2024-05-26T11:50:29.674 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 137/170 2024-05-26T11:50:29.854 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-26T11:50:29.884 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:29.907 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:29.907 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:29.908 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-26T11:50:29.908 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:29.908 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:29.908 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:29.959 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:29.982 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:29.982 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-26T11:50:29.982 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:30.027 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:30.050 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:30.050 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:30.050 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-26T11:50:30.050 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:30.051 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:30.051 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:30.352 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:30.375 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:30.375 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:30.375 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-26T11:50:30.375 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:30.375 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:30.376 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:30.609 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-26T11:50:30.637 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:30.659 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:30.659 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:30.660 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-26T11:50:30.660 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:30.660 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:30.660 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:30.704 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:30.725 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:30.725 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:30.726 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-26T11:50:30.726 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:30.771 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:30.793 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:30.794 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:30.794 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-26T11:50:30.794 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:30.794 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:30.794 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:31.097 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:31.119 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:31.119 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:31.119 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-26T11:50:31.119 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:31.119 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:31.120 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:34.064 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 111/170 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /sys 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /proc 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /mnt 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /var/tmp 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /home 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /root 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout:skipping the directory /tmp 2024-05-26T11:50:34.065 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:34.237 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:34.253 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 112/170 2024-05-26T11:50:34.253 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:34.254 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-26T11:50:34.254 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:34.955 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 143/170 2024-05-26T11:50:34.997 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 144/170 2024-05-26T11:50:35.038 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 145/170 2024-05-26T11:50:35.070 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-26T11:50:35.097 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-26T11:50:35.279 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 148/170 2024-05-26T11:50:35.327 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-26T11:50:35.359 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 150/170 2024-05-26T11:50:35.408 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-26T11:50:35.437 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-26T11:50:35.476 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:35.725 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 143/170 2024-05-26T11:50:35.765 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:35.767 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 144/170 2024-05-26T11:50:35.793 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 113/170 2024-05-26T11:50:35.793 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:35.794 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-26T11:50:35.794 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:35.794 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-26T11:50:35.794 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:35.815 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 145/170 2024-05-26T11:50:35.848 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-26T11:50:35.875 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-26T11:50:36.016 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:36.038 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 114/170 2024-05-26T11:50:36.039 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:36.039 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-26T11:50:36.039 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:36.039 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-26T11:50:36.039 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:36.061 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 148/170 2024-05-26T11:50:36.087 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.105 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-26T11:50:36.131 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.137 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 150/170 2024-05-26T11:50:36.151 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.178 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-26T11:50:36.208 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-26T11:50:36.262 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:36.475 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 115/170 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-26T11:50:36.498 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:36.613 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : automake-1.16.2-8.el9.noarch 116/170 2024-05-26T11:50:36.669 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-ply-3.11-14.el9.noarch 117/170 2024-05-26T11:50:36.726 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 118/170 2024-05-26T11:50:36.844 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.885 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.905 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:36.910 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 119/170 2024-05-26T11:50:36.966 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 120/170 2024-05-26T11:50:36.980 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 155/170 2024-05-26T11:50:37.044 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 121/170 2024-05-26T11:50:37.061 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-26T11:50:37.094 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-26T11:50:37.132 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 122/170 2024-05-26T11:50:37.182 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-26T11:50:37.230 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-26T11:50:37.255 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:37.281 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-26T11:50:37.286 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 123/170 2024-05-26T11:50:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:37.307 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:37.325 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 124/170 2024-05-26T11:50:37.357 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 125/170 2024-05-26T11:50:37.384 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : fuse-2.9.9-15.el9.x86_64 126/170 2024-05-26T11:50:37.392 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:37.424 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-26T11:50:37.424 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:37.518 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:37.542 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-26T11:50:37.548 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:37.548 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:37.727 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:37.728 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:37.765 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 155/170 2024-05-26T11:50:37.803 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:37.828 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-26T11:50:37.845 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:37.849 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-26T11:50:37.874 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 127/170 2024-05-26T11:50:37.889 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-26T11:50:37.892 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:37.906 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 128/170 2024-05-26T11:50:37.968 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 129/170 2024-05-26T11:50:37.970 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-26T11:50:37.997 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:38.002 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-26T11:50:38.027 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:38.052 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-26T11:50:38.079 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:38.079 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:38.172 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:38.213 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-26T11:50:38.213 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:38.297 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:38.323 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-26T11:50:38.327 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:38.328 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:38.507 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:38.508 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:38.600 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:38.625 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-26T11:50:38.688 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:39.039 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 130/170 2024-05-26T11:50:39.082 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:40.053 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 131/170 2024-05-26T11:50:40.083 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:40.194 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 132/170 2024-05-26T11:50:40.361 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 133/170 2024-05-26T11:50:40.385 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:40.409 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 134/170 2024-05-26T11:50:40.410 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:40.410 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-26T11:50:40.410 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:40.410 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-26T11:50:40.410 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:40.461 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:40.473 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 135/170 2024-05-26T11:50:40.575 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 136/170 2024-05-26T11:50:40.633 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 137/170 2024-05-26T11:50:41.554 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libtool-2.4.6-45.el9.x86_64 138/170 2024-05-26T11:50:41.580 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 139/170 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-26T11:50:41.604 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:41.647 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:41.669 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 140/170 2024-05-26T11:50:41.669 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:41.670 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-26T11:50:41.670 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:41.714 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:41.737 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 141/170 2024-05-26T11:50:41.738 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:41.738 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-26T11:50:41.738 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:41.738 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-26T11:50:41.738 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:42.040 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 142/170 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-26T11:50:42.063 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:43.508 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 1/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 3/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 4/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 5/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 6/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 7/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 11/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 12/170 2024-05-26T11:50:43.509 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 13/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 15/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 16/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 17/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 18/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 19/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 20/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 21/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 22/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 23/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 24/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 25/170 2024-05-26T11:50:43.510 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 26/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 27/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 28/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 29/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 30/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 31/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 32/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 33/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 35/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 36/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 37/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 38/170 2024-05-26T11:50:43.511 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 39/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-26T11:50:43.512 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-26T11:50:43.513 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-26T11:50:43.514 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-26T11:50:43.515 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-26T11:50:43.516 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-26T11:50:43.517 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-26T11:50:43.518 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-26T11:50:43.519 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 149/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-26T11:50:43.520 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 151/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-26T11:50:43.521 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-26T11:50:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-26T11:50:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-26T11:50:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-26T11:50:43.522 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 1/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 3/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 4/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 5/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 6/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 7/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:50:44.233 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 11/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 12/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 13/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 15/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 16/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 17/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 18/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 19/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 20/170 2024-05-26T11:50:44.234 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 21/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 22/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 23/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 24/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 25/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 26/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 27/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 28/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 29/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 30/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 31/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 32/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 33/170 2024-05-26T11:50:44.235 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 35/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 36/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 37/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 38/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 39/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-26T11:50:44.236 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-26T11:50:44.237 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-26T11:50:44.238 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-26T11:50:44.239 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-26T11:50:44.240 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-26T11:50:44.241 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-26T11:50:44.242 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-26T11:50:44.243 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-26T11:50:44.244 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 149/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 151/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-26T11:50:44.245 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-26T11:50:44.246 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-26T11:50:46.197 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-26T11:50:46.197 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:46.197 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-05-26T11:50:46.197 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.198 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: automake-1.16.2-8.el9.noarch 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: bison-3.7.4-5.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.199 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.200 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: flex-2.6.4-9.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-26T11:50:46.201 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-26T11:50:46.202 INFO:teuthology.orchestra.run.smithi192.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: lua-5.4.4-4.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-13.14-1.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-26T11:50:46.203 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-26T11:50:46.204 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-26T11:50:46.205 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-26T11:50:46.206 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-26T11:50:46.207 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-26T11:50:46.208 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-26T11:50:46.209 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:50:46.210 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-05-26T11:50:46.628 DEBUG:teuthology.parallel:result is None 2024-05-26T11:50:46.670 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 143/170 2024-05-26T11:50:46.711 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 144/170 2024-05-26T11:50:46.752 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 145/170 2024-05-26T11:50:46.784 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 146/170 2024-05-26T11:50:46.812 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 147/170 2024-05-26T11:50:46.900 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-26T11:50:46.900 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:46.900 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.901 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.902 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.903 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:46.904 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-26T11:50:46.905 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.906 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: lua-5.4.4-4.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-26T11:50:46.907 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.14-1.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-26T11:50:46.908 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-26T11:50:46.909 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-26T11:50:46.910 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:46.911 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-26T11:50:46.912 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-26T11:50:46.913 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-26T11:50:46.914 INFO:teuthology.orchestra.run.smithi186.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-26T11:50:46.915 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-26T11:50:46.916 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-26T11:50:46.916 INFO:teuthology.orchestra.run.smithi186.stdout: 2024-05-26T11:50:46.916 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2024-05-26T11:50:47.013 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 148/170 2024-05-26T11:50:47.067 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 149/170 2024-05-26T11:50:47.099 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 150/170 2024-05-26T11:50:47.132 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 151/170 2024-05-26T11:50:47.161 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-team-1:1.47.90-1.el9.x86_64 152/170 2024-05-26T11:50:47.207 INFO:teuthology.orchestra.run.smithi156.stdout: Upgrading : NetworkManager-tui-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:47.355 DEBUG:teuthology.parallel:result is None 2024-05-26T11:50:47.805 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:47.849 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:47.867 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 154/170 2024-05-26T11:50:48.698 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 155/170 2024-05-26T11:50:48.795 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 156/170 2024-05-26T11:50:48.836 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 157/170 2024-05-26T11:50:48.942 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : flex-2.6.4-9.el9.x86_64 158/170 2024-05-26T11:50:48.973 INFO:teuthology.orchestra.run.smithi156.stdout: Installing : bison-3.7.4-5.el9.x86_64 159/170 2024-05-26T11:50:48.999 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:49.023 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 161/170 2024-05-26T11:50:49.050 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:49.050 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:49.143 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 163/170 2024-05-26T11:50:49.167 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 164/170 2024-05-26T11:50:49.167 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:49.252 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 165/170 2024-05-26T11:50:49.277 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 166/170 2024-05-26T11:50:49.290 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:49.291 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:49.479 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 167/170 2024-05-26T11:50:49.479 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:49.564 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 168/170 2024-05-26T11:50:49.596 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 169/170 2024-05-26T11:50:49.662 INFO:teuthology.orchestra.run.smithi156.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:55.220 INFO:teuthology.orchestra.run.smithi156.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 170/170 2024-05-26T11:50:55.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 1/170 2024-05-26T11:50:55.220 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 3/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 4/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5 5/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 6/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 7/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 8/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 9/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 10/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 11/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 12/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 13/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 14/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_ 15/170 2024-05-26T11:50:55.221 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 16/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 17/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_6 18/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86 19/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 20/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el 21/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9. 22/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_6 23/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 24/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 25/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 26/170 2024-05-26T11:50:55.222 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 27/170 2024-05-26T11:50:55.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 28/170 2024-05-26T11:50:55.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 29/170 2024-05-26T11:50:55.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853. 30/170 2024-05-26T11:50:55.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noa 31/170 2024-05-26T11:50:55.234 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.n 32/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4 33/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el 34/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 35/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.e 36/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 37/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 38/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 39/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/170 2024-05-26T11:50:55.235 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 52/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/170 2024-05-26T11:50:55.236 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/170 2024-05-26T11:50:55.237 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 76/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 77/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 78/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 79/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 80/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 81/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 82/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 83/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 84/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 85/170 2024-05-26T11:50:55.238 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 86/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 87/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 88/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 89/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 90/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 91/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 92/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 93/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 94/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 95/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 96/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 97/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 98/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 99/170 2024-05-26T11:50:55.239 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 100/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 101/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 102/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 103/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 104/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 105/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 106/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 107/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 108/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 109/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 110/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 111/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 112/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 113/170 2024-05-26T11:50:55.240 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 114/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 115/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 116/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 117/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 118/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 119/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 120/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 121/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/170 2024-05-26T11:50:55.241 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/170 2024-05-26T11:50:55.242 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 149/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 151/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.47.90-1.el9.x86_64 153/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.47.90-1.el9.x86_64 155/170 2024-05-26T11:50:55.243 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.47.90-1.el9.x86_64 157/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.47.90-1.el9.x86_64 159/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 161/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 162/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 163/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 164/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 165/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 166/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 167/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 168/170 2024-05-26T11:50:55.244 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 169/170 2024-05-26T11:50:57.949 INFO:teuthology.orchestra.run.smithi156.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 170/170 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout:Upgraded: 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-libnm-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-team-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: NetworkManager-tui-1:1.47.90-1.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-05-26T11:50:57.950 INFO:teuthology.orchestra.run.smithi156.stdout: librados2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: librbd1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout:Installed: 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: automake-1.16.2-8.el9.noarch 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: bison-3.7.4-5.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-base-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.951 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-grafana-dashboards-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-immutable-object-cache-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mds-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-dashboard-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-modules-core-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mgr-rook-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-mon-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-osd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-prometheus-alerts-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-radosgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.952 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-selinux-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-test-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: ceph-volume-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: cephadm-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-shell-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: cephfs-top-2:19.0.0-3867.g9a4c5853.el9.noarch 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: flex-2.6.4-9.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-26T11:50:57.953 INFO:teuthology.orchestra.run.smithi156.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libcephfs2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libcephsqlite-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-26T11:50:57.954 INFO:teuthology.orchestra.run.smithi156.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: librados-devel-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: libradosstriper1-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: librgw2-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: lua-5.4.4-4.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: luarocks-3.9.2-5.el9.noarch 2024-05-26T11:50:57.955 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-13.14-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-26T11:50:57.956 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-argparse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ceph-common-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cephfs-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-26T11:50:57.957 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-26T11:50:57.958 INFO:teuthology.orchestra.run.smithi156.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-ply-3.11-14.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-26T11:50:57.959 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rados-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rgw-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-26T11:50:57.960 INFO:teuthology.orchestra.run.smithi156.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-26T11:50:57.961 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-fuse-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-mirror-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: rbd-nbd-2:19.0.0-3867.g9a4c5853.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: re2-1:20211101-3.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-26T11:50:57.962 INFO:teuthology.orchestra.run.smithi156.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-26T11:50:57.963 INFO:teuthology.orchestra.run.smithi156.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-26T11:50:57.963 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:50:57.963 INFO:teuthology.orchestra.run.smithi156.stdout:Complete! 2024-05-26T11:50:58.425 DEBUG:teuthology.parallel:result is None 2024-05-26T11:50:58.426 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:50:58.781 DEBUG:teuthology.orchestra.run.smithi156:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-26T11:50:58.803 INFO:teuthology.orchestra.run.smithi156.stdout:19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:58.804 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:58.804 INFO:teuthology.task.install:The correct ceph version 19.0.0-3867.g9a4c5853 is installed. 2024-05-26T11:50:58.806 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:50:59.133 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-26T11:50:59.159 INFO:teuthology.orchestra.run.smithi186.stdout:19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:59.159 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:59.159 INFO:teuthology.task.install:The correct ceph version 19.0.0-3867.g9a4c5853 is installed. 2024-05-26T11:50:59.161 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:50:59.468 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-26T11:50:59.494 INFO:teuthology.orchestra.run.smithi192.stdout:19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:59.494 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3867.g9a4c5853.el9 2024-05-26T11:50:59.494 INFO:teuthology.task.install:The correct ceph version 19.0.0-3867.g9a4c5853 is installed. 2024-05-26T11:50:59.496 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-26T11:50:59.496 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:50:59.497 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-26T11:50:59.529 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:50:59.529 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-26T11:50:59.561 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:50:59.561 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-26T11:50:59.592 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-26T11:50:59.593 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:50:59.593 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/daemon-helper 2024-05-26T11:50:59.621 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-26T11:50:59.688 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:50:59.688 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2024-05-26T11:50:59.716 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-26T11:50:59.784 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:50:59.784 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-05-26T11:50:59.810 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-26T11:50:59.873 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-26T11:50:59.873 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:50:59.873 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-26T11:50:59.901 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-26T11:50:59.968 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:50:59.968 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-26T11:50:59.999 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-26T11:51:00.064 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:51:00.064 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-26T11:51:00.088 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-26T11:51:00.151 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-26T11:51:00.152 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:51:00.152 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/usr/bin/stdin-killer 2024-05-26T11:51:00.179 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-26T11:51:00.246 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:51:00.246 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2024-05-26T11:51:00.270 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-26T11:51:00.334 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:51:00.334 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-05-26T11:51:00.357 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-26T11:51:00.421 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-26T11:51:00.512 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy'], 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-26T11:51:00.512 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:51:00.513 INFO:tasks.cephadm:Cluster fsid is 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:51:00.513 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-26T11:51:00.513 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.156', 'mon.b': '172.21.15.186', 'mon.c': '172.21.15.192'} 2024-05-26T11:51:00.513 INFO:tasks.cephadm:First mon is mon.a on smithi156 2024-05-26T11:51:00.513 INFO:tasks.cephadm:First mgr is x 2024-05-26T11:51:00.513 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-26T11:51:00.513 DEBUG:teuthology.orchestra.run.smithi156:> sudo hostname $(hostname -s) 2024-05-26T11:51:00.544 DEBUG:teuthology.orchestra.run.smithi186:> sudo hostname $(hostname -s) 2024-05-26T11:51:00.568 DEBUG:teuthology.orchestra.run.smithi192:> sudo hostname $(hostname -s) 2024-05-26T11:51:00.591 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-26T11:51:00.591 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:51:00.932 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/79709/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3867-g9a4c5853', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3867.g9a4c5853'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-26 10:19:48.916867', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-lusov-quiesce', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-05-26T11:51:01.072 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref wip-lusov-quiesce, sha1 9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:51:01.073 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/x86_64/flavors/default/cephadm 2024-05-26T11:51:01.073 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/x86_64/flavors/default/cephadm 2024-05-26T11:51:01.073 DEBUG:teuthology.orchestra.run.smithi156:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:01.995 INFO:teuthology.orchestra.run.smithi156.stdout:-rw-r--r--. 1 ubuntu ubuntu 782874 May 26 11:51 /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:01.996 DEBUG:teuthology.orchestra.run.smithi186:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:02.468 INFO:teuthology.orchestra.run.smithi186.stdout:-rw-r--r--. 1 ubuntu ubuntu 782874 May 26 11:51 /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:02.468 DEBUG:teuthology.orchestra.run.smithi192:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/wip-lusov-quiesce/9a4c5853d1c2a353f72cd6358bbdedd93c4cc209/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:03.449 INFO:teuthology.orchestra.run.smithi192.stdout:-rw-r--r--. 1 ubuntu ubuntu 782874 May 26 11:51 /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:03.449 DEBUG:teuthology.orchestra.run.smithi156:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:03.469 DEBUG:teuthology.orchestra.run.smithi186:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:03.488 DEBUG:teuthology.orchestra.run.smithi192:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-26T11:51:03.522 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 on all hosts... 2024-05-26T11:51:03.522 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 pull 2024-05-26T11:51:03.525 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 pull 2024-05-26T11:51:03.530 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 pull 2024-05-26T11:51:04.355 INFO:teuthology.orchestra.run.smithi192.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209... 2024-05-26T11:51:04.440 INFO:teuthology.orchestra.run.smithi186.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209... 2024-05-26T11:51:04.476 INFO:teuthology.orchestra.run.smithi156.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209... 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout:{ 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: "ceph_version": "ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev)", 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: "image_id": "939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409", 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: "repo_digests": [ 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b" 2024-05-26T11:51:34.932 INFO:teuthology.orchestra.run.smithi192.stdout: ] 2024-05-26T11:51:34.933 INFO:teuthology.orchestra.run.smithi192.stdout:} 2024-05-26T11:51:56.117 INFO:teuthology.orchestra.run.smithi186.stdout:{ 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout: "ceph_version": "ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev)", 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout: "image_id": "939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409", 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout: "repo_digests": [ 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b" 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout: ] 2024-05-26T11:51:56.118 INFO:teuthology.orchestra.run.smithi186.stdout:} 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout: "ceph_version": "ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev)", 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout: "image_id": "939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409", 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout: "repo_digests": [ 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b" 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout: ] 2024-05-26T11:51:58.171 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T11:51:58.203 DEBUG:teuthology.orchestra.run.smithi156:> sudo mkdir -p /etc/ceph 2024-05-26T11:51:58.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph 2024-05-26T11:51:58.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph 2024-05-26T11:51:58.312 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 777 /etc/ceph 2024-05-26T11:51:58.336 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /etc/ceph 2024-05-26T11:51:58.363 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /etc/ceph 2024-05-26T11:51:58.391 INFO:tasks.cephadm:Writing seed config... 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-26T11:51:58.392 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-26T11:51:58.393 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-26T11:51:58.394 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-26T11:51:58.395 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-26T11:51:58.396 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:51:58.396 DEBUG:teuthology.orchestra.run.smithi156:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-26T11:51:58.411 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 38c1aff8-1b56-11ef-bc9b-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-26T11:51:58.411 DEBUG:teuthology.orchestra.run.smithi156:mon.a> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service 2024-05-26T11:51:58.453 DEBUG:teuthology.orchestra.run.smithi156:mgr.x> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x.service 2024-05-26T11:51:58.496 INFO:tasks.cephadm:Bootstrapping... 2024-05-26T11:51:58.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 -v bootstrap --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.156 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-26T11:51:58.690 INFO:teuthology.orchestra.run.smithi156.stdout:-------------------------------------------------------------------------------- 2024-05-26T11:51:58.690 INFO:teuthology.orchestra.run.smithi156.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209', '-v', 'bootstrap', '--fsid', '38c1aff8-1b56-11ef-bc9b-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.156', '--skip-admin-label'] 2024-05-26T11:51:58.690 INFO:teuthology.orchestra.run.smithi156.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-26T11:51:58.690 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying podman|docker is present... 2024-05-26T11:51:58.718 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stdout 5.0.2 2024-05-26T11:51:58.718 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying lvm2 is present... 2024-05-26T11:51:58.718 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying time synchronization is in place... 2024-05-26T11:51:58.725 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-26T11:51:58.725 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-26T11:51:58.731 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-26T11:51:58.732 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:51:58.738 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout enabled 2024-05-26T11:51:58.745 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout active 2024-05-26T11:51:58.745 INFO:teuthology.orchestra.run.smithi156.stdout:Unit chronyd.service is enabled and running 2024-05-26T11:51:58.745 INFO:teuthology.orchestra.run.smithi156.stdout:Repeating the final host check... 2024-05-26T11:51:58.769 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stdout 5.0.2 2024-05-26T11:51:58.769 INFO:teuthology.orchestra.run.smithi156.stdout:podman (/bin/podman) version 5.0.2 is present 2024-05-26T11:51:58.769 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl is present 2024-05-26T11:51:58.769 INFO:teuthology.orchestra.run.smithi156.stdout:lvcreate is present 2024-05-26T11:51:58.776 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-26T11:51:58.776 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-26T11:51:58.782 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-26T11:51:58.783 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:51:58.789 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout enabled 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout active 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Unit chronyd.service is enabled and running 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Host looks OK 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Cluster fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Acquiring lock 139800755637792 on /run/cephadm/38c1aff8-1b56-11ef-bc9b-c7b262605968.lock 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Lock 139800755637792 acquired on /run/cephadm/38c1aff8-1b56-11ef-bc9b-c7b262605968.lock 2024-05-26T11:51:58.796 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying IP 172.21.15.156 port 3300 ... 2024-05-26T11:51:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying IP 172.21.15.156 port 6789 ... 2024-05-26T11:51:58.797 INFO:teuthology.orchestra.run.smithi156.stdout:Base mon IP(s) is [172.21.15.156:3300, 172.21.15.156:6789], mon addrv is [v2:172.21.15.156:3300,v1:172.21.15.156:6789] 2024-05-26T11:51:58.800 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.156 metric 100 2024-05-26T11:51:58.800 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.156 metric 100 2024-05-26T11:51:58.802 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-26T11:51:58.802 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-26T11:51:58.802 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1790sec hoplimit 64 pref medium 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:6dfd/64 scope link 2024-05-26T11:51:58.805 INFO:teuthology.orchestra.run.smithi156.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-26T11:51:58.806 INFO:teuthology.orchestra.run.smithi156.stdout:Mon IP `172.21.15.156` is in CIDR network `172.21.0.0/20` 2024-05-26T11:51:58.806 INFO:teuthology.orchestra.run.smithi156.stdout:Mon IP `172.21.15.156` is in CIDR network `172.21.0.0/20` 2024-05-26T11:51:58.806 INFO:teuthology.orchestra.run.smithi156.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-26T11:51:58.806 INFO:teuthology.orchestra.run.smithi156.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-26T11:51:58.807 INFO:teuthology.orchestra.run.smithi156.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209... 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stdout 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209... 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Getting image source signatures 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Copying blob sha256:beb11317ad99bbfd94ff5e7f30ef64ece0d4bd61ee9f3a7f6ade70a7758bb5a1 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Copying blob sha256:6ac36e7c57f227110df5cd65cfdca1cc05e0b17df6748637a6aa04c5c80f89c6 2024-05-26T11:51:59.607 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Copying config sha256:939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 2024-05-26T11:51:59.608 INFO:teuthology.orchestra.run.smithi156.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-26T11:52:00.504 INFO:teuthology.orchestra.run.smithi156.stdout:ceph: stdout ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev) 2024-05-26T11:52:00.504 INFO:teuthology.orchestra.run.smithi156.stdout:Ceph version: ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev) 2024-05-26T11:52:00.504 INFO:teuthology.orchestra.run.smithi156.stdout:Extracting ceph user uid/gid from container image... 2024-05-26T11:52:01.485 INFO:teuthology.orchestra.run.smithi156.stdout:stat: stdout 167 167 2024-05-26T11:52:01.485 INFO:teuthology.orchestra.run.smithi156.stdout:Creating initial keys... 2024-05-26T11:52:02.524 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph-authtool: stdout AQBhIlNmuE1yMBAAl4SjJi90uYHAyQ7hVXKZtA== 2024-05-26T11:52:03.531 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph-authtool: stdout AQBiIlNmE+dUNhAAbtGYczkyRBGkv//oYGUSdw== 2024-05-26T11:52:04.378 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph-authtool: stdout AQBjIlNm6QysNBAAMNsJfJpwI2eYgzvSyKDRFw== 2024-05-26T11:52:04.379 INFO:teuthology.orchestra.run.smithi156.stdout:Creating initial monmap... 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:monmaptool for a [v2:172.21.15.156:3300,v1:172.21.15.156:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:setting min_mon_release = quincy 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: set fsid to 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:05.235 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-26T11:52:05.236 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:52:05.236 INFO:teuthology.orchestra.run.smithi156.stdout:Creating mon... 2024-05-26T11:52:06.075 INFO:teuthology.orchestra.run.smithi156.stdout:create mon.a on 2024-05-26T11:52:06.269 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-26T11:52:06.418 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-26T11:52:06.609 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target. 2024-05-26T11:52:06.609 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target → /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target. 2024-05-26T11:52:06.840 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a 2024-05-26T11:52:06.840 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Failed to reset failed state of unit ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service: Unit ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service not loaded. 2024-05-26T11:52:07.006 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target.wants/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service → /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@.service. 2024-05-26T11:52:07.584 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-26T11:52:07.584 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout disabled 2024-05-26T11:52:07.591 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-26T11:52:07.591 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:52:07.591 INFO:teuthology.orchestra.run.smithi156.stdout:firewalld.service is not enabled 2024-05-26T11:52:07.591 INFO:teuthology.orchestra.run.smithi156.stdout:Not possible to enable service . firewalld.service is not available 2024-05-26T11:52:07.591 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mon to start... 2024-05-26T11:52:07.592 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mon... 2024-05-26T11:52:07.716 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:07 smithi156 systemd[1]: Started Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:52:08.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-26T11:52:08.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: monmap epoch 1 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: last_changed 2024-05-26T11:52:04.693642+0000 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: min_mon_release 19 (squid) 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: election_strategy: 1 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: fsmap 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: osdmap e1: 0 total, 0 up, 0 in 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: mgrmap e1: no daemons active 2024-05-26T11:52:08.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:08 smithi156 ceph-mon[31654]: from='client.? 172.21.15.156:0/2896261854' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-26T11:52:08.684 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout cluster: 2024-05-26T11:52:08.684 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout id: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout services: 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.54486s) 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout data: 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout pgs: 2024-05-26T11:52:08.685 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.004+0000 7f92ad4c8700 1 Processor -- start 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.004+0000 7f92ad4c8700 1 -- start start 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.004+0000 7f92ad4c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.004+0000 7f92ad4c8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92a81098e0 con 0x7f92a8108f80 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58676/0 (socket says 172.21.15.156:58676) 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a6ffd700 1 -- 172.21.15.156:0/4216509880 learned_addr learned my addr 172.21.15.156:0/4216509880 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a6ffd700 1 -- 172.21.15.156:0/4216509880 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a810a090 con 0x7f92a8108f80 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a6ffd700 1 --2- 172.21.15.156:0/4216509880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f92980098a0 tx=0x7f9298009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6c5abef3c22ae86 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a67fc700 1 -- 172.21.15.156:0/4216509880 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9298004160 con 0x7f92a8108f80 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a67fc700 1 -- 172.21.15.156:0/4216509880 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9298004300 con 0x7f92a8108f80 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.005+0000 7f92a67fc700 1 -- 172.21.15.156:0/4216509880 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92980036a0 con 0x7f92a8108f80 2024-05-26T11:52:08.686 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 -- 172.21.15.156:0/4216509880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 msgr2=0x7f92a81093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 --2- 172.21.15.156:0/4216509880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f92980098a0 tx=0x7f9298009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 -- 172.21.15.156:0/4216509880 shutdown_connections 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 --2- 172.21.15.156:0/4216509880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a81093a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 -- 172.21.15.156:0/4216509880 >> 172.21.15.156:0/4216509880 conn(0x7f92a81045a0 msgr2=0x7f92a81069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 -- 172.21.15.156:0/4216509880 shutdown_connections 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.006+0000 7f92ad4c8700 1 -- 172.21.15.156:0/4216509880 wait complete. 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92ad4c8700 1 Processor -- start 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92ad4c8700 1 -- start start 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92ad4c8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92ad4c8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f92a810a560 con 0x7f92a8108f80 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92a6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58686/0 (socket says 172.21.15.156:58686) 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92a6ffd700 1 -- 172.21.15.156:0/2896261854 learned_addr learned my addr 172.21.15.156:0/2896261854 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.007+0000 7f92a6ffd700 1 -- 172.21.15.156:0/2896261854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92a807f8f0 con 0x7f92a8108f80 2024-05-26T11:52:08.687 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a6ffd700 1 --2- 172.21.15.156:0/2896261854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9298000c00 tx=0x7f9298004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9298003db0 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9298011630 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9298011950 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92a807db30 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92a807e010 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f929801a460 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.008+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f929801b780 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.009+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92a8064ba0 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.011+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f929801b980 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.080+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f92a8195d90 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.080+0000 7f92a4ff9700 1 -- 172.21.15.156:0/2896261854 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+316 (secure 0 0 0) 0x7f929801bb60 con 0x7f92a8108f80 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 msgr2=0x7f92a807f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 --2- 172.21.15.156:0/2896261854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f9298000c00 tx=0x7f9298004750 comp rx=0 tx=0).stop 2024-05-26T11:52:08.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 shutdown_connections 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 --2- 172.21.15.156:0/2896261854 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f92a8108f80 0x7f92a807f3b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 >> 172.21.15.156:0/2896261854 conn(0x7f92a81045a0 msgr2=0x7f92a81931e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 shutdown_connections 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:08.081+0000 7f92ad4c8700 1 -- 172.21.15.156:0/2896261854 wait complete. 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:mon is available 2024-05-26T11:52:08.689 INFO:teuthology.orchestra.run.smithi156.stdout:Assimilating anything we can from ceph.conf... 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [global] 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout fsid = 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.156:3300,v1:172.21.15.156:6789] 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-26T11:52:09.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [osd] 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f4750876700 1 Processor -- start 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f4750876700 1 -- start start 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f4750876700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:09.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f4750876700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f474c07e1a0 con 0x7f474c07f720 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f474a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f474a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58690/0 (socket says 172.21.15.156:58690) 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.079+0000 7f474a59c700 1 -- 172.21.15.156:0/3492045025 learned_addr learned my addr 172.21.15.156:0/3492045025 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f474a59c700 1 -- 172.21.15.156:0/3492045025 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c07e320 con 0x7f474c07f720 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f474a59c700 1 --2- 172.21.15.156:0/3492045025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f473c0098a0 tx=0x7f473c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd1437e2c074c3e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f474959a700 1 -- 172.21.15.156:0/3492045025 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f473c004160 con 0x7f474c07f720 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f474959a700 1 -- 172.21.15.156:0/3492045025 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f473c004300 con 0x7f474c07f720 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f474959a700 1 -- 172.21.15.156:0/3492045025 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f473c0036a0 con 0x7f474c07f720 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f4750876700 1 -- 172.21.15.156:0/3492045025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 msgr2=0x7f474c07dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.080+0000 7f4750876700 1 --2- 172.21.15.156:0/3492045025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f473c0098a0 tx=0x7f473c009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 -- 172.21.15.156:0/3492045025 shutdown_connections 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 --2- 172.21.15.156:0/3492045025 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c07dbd0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:09.694 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 -- 172.21.15.156:0/3492045025 >> 172.21.15.156:0/3492045025 conn(0x7f474c104560 msgr2=0x7f474c106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 -- 172.21.15.156:0/3492045025 shutdown_connections 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 -- 172.21.15.156:0/3492045025 wait complete. 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 Processor -- start 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.081+0000 7f4750876700 1 -- start start 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f4750876700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f4750876700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f474c07e850 con 0x7f474c07f720 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f474a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f474a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58694/0 (socket says 172.21.15.156:58694) 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f474a59c700 1 -- 172.21.15.156:0/4225972723 learned_addr learned my addr 172.21.15.156:0/4225972723 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f474a59c700 1 -- 172.21.15.156:0/4225972723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f474c1a31a0 con 0x7f474c07f720 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f474a59c700 1 --2- 172.21.15.156:0/4225972723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:09.695 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f473c003db0 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f473c011630 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f473c011950 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f474c1a3430 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f474c1a6120 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f473c01a460 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.082+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f473c01b780 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.083+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f474c064ba0 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.085+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f473c01ba20 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.152+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f474c02e790 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.155+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+467 (secure 0 0 0) 0x7f473c01bc00 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.155+0000 7f473b7fe700 1 -- 172.21.15.156:0/4225972723 <== mon.0 v2:172.21.15.156:3300/0 8 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f473c01bde0 con 0x7f474c07f720 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 msgr2=0x7f474c1a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:09.696 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 --2- 172.21.15.156:0/4225972723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f473c000c00 tx=0x7f473c004750 comp rx=0 tx=0).stop 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 shutdown_connections 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 --2- 172.21.15.156:0/4225972723 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f474c07f720 0x7f474c1a2c60 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 >> 172.21.15.156:0/4225972723 conn(0x7f474c104560 msgr2=0x7f474c106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 shutdown_connections 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:09.156+0000 7f4750876700 1 -- 172.21.15.156:0/4225972723 wait complete. 2024-05-26T11:52:09.697 INFO:teuthology.orchestra.run.smithi156.stdout:Generating new minimal ceph.conf... 2024-05-26T11:52:09.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:09 smithi156 ceph-mon[31654]: from='client.? 172.21.15.156:0/4225972723' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-26T11:52:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:09 smithi156 ceph-mon[31654]: from='client.? 172.21.15.156:0/4225972723' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-26T11:52:10.749 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:10 smithi156 ceph-mon[31654]: from='client.? 172.21.15.156:0/1956773829' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.094+0000 7f7ce9326700 1 Processor -- start 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.094+0000 7f7ce9326700 1 -- start start 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.094+0000 7f7ce9326700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.094+0000 7f7ce9326700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce41098e0 con 0x7f7ce4108f80 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58706/0 (socket says 172.21.15.156:58706) 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce2ffd700 1 -- 172.21.15.156:0/242775634 learned_addr learned my addr 172.21.15.156:0/242775634 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce2ffd700 1 -- 172.21.15.156:0/242775634 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce410a090 con 0x7f7ce4108f80 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce2ffd700 1 --2- 172.21.15.156:0/242775634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40098a0 tx=0x7f7cd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af6dd8176b573e6e server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce27fc700 1 -- 172.21.15.156:0/242775634 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd4004160 con 0x7f7ce4108f80 2024-05-26T11:52:10.755 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce27fc700 1 -- 172.21.15.156:0/242775634 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7cd4004300 con 0x7f7ce4108f80 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.095+0000 7f7ce27fc700 1 -- 172.21.15.156:0/242775634 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd4011af0 con 0x7f7ce4108f80 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 -- 172.21.15.156:0/242775634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 msgr2=0x7f7ce41093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 --2- 172.21.15.156:0/242775634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40098a0 tx=0x7f7cd4009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 -- 172.21.15.156:0/242775634 shutdown_connections 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 --2- 172.21.15.156:0/242775634 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce41093a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 -- 172.21.15.156:0/242775634 >> 172.21.15.156:0/242775634 conn(0x7f7ce4104560 msgr2=0x7f7ce4106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 -- 172.21.15.156:0/242775634 shutdown_connections 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 -- 172.21.15.156:0/242775634 wait complete. 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.096+0000 7f7ce9326700 1 Processor -- start 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce9326700 1 -- start start 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce9326700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce9326700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7ce410d4d0 con 0x7f7ce4108f80 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58718/0 (socket says 172.21.15.156:58718) 2024-05-26T11:52:10.756 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce2ffd700 1 -- 172.21.15.156:0/1956773829 learned_addr learned my addr 172.21.15.156:0/1956773829 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce2ffd700 1 -- 172.21.15.156:0/1956773829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ce419d420 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce2ffd700 1 --2- 172.21.15.156:0/1956773829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fb0 tx=0x7f7cd4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd40036a0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7cd401a430 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ce419d6b0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7cd4019550 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.097+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ce41a03a0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.098+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7cd4019b90 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.098+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7cd4024730 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.099+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ce4064ba0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.101+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f7cd4043ec0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.169+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f7ce41a08f0 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.169+0000 7f7ce0ff9700 1 -- 172.21.15.156:0/1956773829 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+177 (secure 0 0 0) 0x7f7cd4032030 con 0x7f7ce4108f80 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 msgr2=0x7f7ce419cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:10.757 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 --2- 172.21.15.156:0/1956773829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fb0 tx=0x7f7cd4011940 comp rx=0 tx=0).stop 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 shutdown_connections 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 --2- 172.21.15.156:0/1956773829 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7ce4108f80 0x7f7ce419cee0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 >> 172.21.15.156:0/1956773829 conn(0x7f7ce4104560 msgr2=0x7f7ce41050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 shutdown_connections 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:10.170+0000 7f7ce9326700 1 -- 172.21.15.156:0/1956773829 wait complete. 2024-05-26T11:52:10.758 INFO:teuthology.orchestra.run.smithi156.stdout:Restarting the monitor... 2024-05-26T11:52:11.003 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:10 smithi156 systemd[1]: Stopping Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:52:11.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a[31650]: 2024-05-26T11:52:11.001+0000 7efe15daa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T11:52:11.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a[31650]: 2024-05-26T11:52:11.001+0000 7efe15daa700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-26T11:52:11.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 podman[31898]: 2024-05-26 11:52:11.145724956 +0000 UTC m=+0.227408753 container died 528b0d8397301fbc5ab7077630003e069639768c8b95425f3de7ec57c90c9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-26T11:52:11.744 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 podman[31898]: 2024-05-26 11:52:11.493736751 +0000 UTC m=+0.575420539 container cleanup 528b0d8397301fbc5ab7077630003e069639768c8b95425f3de7ec57c90c9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-26T11:52:11.745 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 bash[31898]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a 2024-05-26T11:52:12.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 podman[31910]: 2024-05-26 11:52:11.744732379 +0000 UTC m=+0.598501742 container remove 528b0d8397301fbc5ab7077630003e069639768c8b95425f3de7ec57c90c9aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-26T11:52:12.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-26T11:52:12.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 systemd[1]: Stopped Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:52:12.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:11 smithi156 systemd[1]: Starting Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:52:12.391 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 podman[32006]: 2024-05-26 11:52:12.029612484 +0000 UTC m=+0.017586998 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 2024-05-26T11:52:12.392 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 podman[32006]: 2024-05-26 11:52:12.154271005 +0000 UTC m=+0.142245559 container create 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-26T11:52:12.392 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 podman[32006]: 2024-05-26 11:52:12.365156997 +0000 UTC m=+0.353131509 container init 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-26T11:52:12.392 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 podman[32006]: 2024-05-26 11:52:12.368420954 +0000 UTC m=+0.356395466 container start 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T11:52:12.392 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: set uid:gid to 167:167 (ceph:ceph) 2024-05-26T11:52:12.436 INFO:teuthology.orchestra.run.smithi156.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-26T11:52:12.644 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: ceph version 19.0.0-3867-g9a4c5853 (9a4c5853d1c2a353f72cd6358bbdedd93c4cc209) squid (dev), process ceph-mon, pid 2 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: pidfile_write: ignore empty --pid-file 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: load: jerasure load: lrc 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: RocksDB version: 7.9.2 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Git sha 0 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Compile date 2024-05-26 08:57:40 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: DB SUMMARY 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: DB Session ID: BMQMZAO7FOEZJAH6NYOH 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: CURRENT file: CURRENT 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: IDENTITY file: IDENTITY 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105240 ; 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.error_if_exists: 0 2024-05-26T11:52:12.645 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.create_if_missing: 0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.paranoid_checks: 1 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.env: 0x55d17ce11cc0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.fs: PosixFileSystem 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.info_log: 0x55d17dc6dda0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_file_opening_threads: 16 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.statistics: (nil) 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.use_fsync: 0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_log_file_size: 0 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-26T11:52:12.646 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.keep_log_file_num: 1000 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.recycle_log_file_num: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_fallocate: 1 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_mmap_reads: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_mmap_writes: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.use_direct_reads: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.create_missing_column_families: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.db_log_dir: 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.wal_dir: 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-26T11:52:12.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.advise_random_on_open: 1 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.db_write_buffer_size: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.write_buffer_manager: 0x55d17dc3f0e0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.rate_limiter: (nil) 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.wal_recovery_mode: 2 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enable_thread_tracking: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enable_pipelined_write: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.unordered_write: 0 2024-05-26T11:52:12.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.row_cache: None 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.wal_filter: None 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_ingest_behind: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.two_write_queues: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.manual_wal_flush: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.wal_compression: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.atomic_flush: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-26T11:52:12.649 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.log_readahead_size: 0 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.best_efforts_recovery: 0 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.allow_data_in_errors: 0 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.db_host_id: __hostname__ 2024-05-26T11:52:12.650 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-26T11:52:12.651 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_background_jobs: 2 2024-05-26T11:52:12.651 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_background_compactions: -1 2024-05-26T11:52:12.651 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_subcompactions: 1 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_total_wal_size: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_open_files: -1 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bytes_per_sync: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_readahead_size: 0 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_background_flushes: -1 2024-05-26T11:52:12.652 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Compression algorithms supported: 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kZSTD supported: 0 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kXpressCompression supported: 0 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kLZ4HCCompression supported: 1 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kZlibCompression supported: 1 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kSnappyCompression supported: 1 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kLZ4Compression supported: 1 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: kBZip2Compression supported: 0 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.merge_operator: 2024-05-26T11:52:12.653 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_filter: None 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_filter_factory: None 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.sst_partitioner_factory: None 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55d17dc6df60) 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: cache_index_and_filter_blocks: 1 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: pin_top_level_index_and_filter: 1 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: index_type: 0 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: data_block_index_type: 0 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: index_shortening: 1 2024-05-26T11:52:12.654 INFO:journalctl@ceph.mon.a.smithi156.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: checksum: 4 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: no_block_cache: 0 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_cache: 0x55d17dc711f0 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_cache_name: BinnedLRUCache 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_cache_options: 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: capacity : 536870912 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: num_shard_bits : 4 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: strict_capacity_limit : 0 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: high_pri_pool_ratio: 0.000 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_cache_compressed: (nil) 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: persistent_cache: (nil) 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_size: 4096 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_size_deviation: 10 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_restart_interval: 16 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: index_block_restart_interval: 1 2024-05-26T11:52:12.655 INFO:journalctl@ceph.mon.a.smithi156.stdout: metadata_block_size: 4096 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: partition_filters: 0 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: use_delta_encoding: 1 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: filter_policy: bloomfilter 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: whole_key_filtering: 1 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: verify_compression: 0 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: read_amp_bytes_per_bit: 0 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: format_version: 5 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: enable_index_compression: 1 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: block_align: 0 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: max_auto_readahead_size: 262144 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: prepopulate_block_cache: 0 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: initial_auto_readahead_size: 8192 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout: num_file_reads_for_auto_readahead: 2 2024-05-26T11:52:12.656 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.write_buffer_size: 33554432 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_write_buffer_number: 2 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression: NoCompression 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression: Disabled 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.prefix_extractor: nullptr 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.num_levels: 7 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-26T11:52:12.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.level: 32767 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.strategy: 0 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.enabled: false 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-26T11:52:12.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.target_file_size_base: 67108864 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-26T11:52:12.659 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.arena_block_size: 1048576 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.disable_auto_compactions: 0 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-26T11:52:12.660 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.inplace_update_support: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.bloom_locality: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.max_successive_merges: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.paranoid_file_checks: 0 2024-05-26T11:52:12.661 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.force_consistency_checks: 1 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.report_bg_io_stats: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.ttl: 2592000 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enable_blob_files: false 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.min_blob_size: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_file_size: 268435456 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-26T11:52:12.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.blob_file_starting_level: 0 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 2f171602-63a0-43e3-bcd6-c79c03f2cfb1 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716724332398017, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716724332399000, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100391, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98285, "index_size": 298, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14191, "raw_average_key_size": 49, "raw_value_size": 90554, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1716724332, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "2f171602-63a0-43e3-bcd6-c79c03f2cfb1", "db_session_id": "BMQMZAO7FOEZJAH6NYOH", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: EVENT_LOG_v1 {"time_micros": 1716724332399081, "job": 1, "event": "recovery_finished"} 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55d17dd8e000 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: DB pointer 0x55d17dd7a000 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-26T11:52:12.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: ** DB Stats ** 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: ** Compaction Stats [default] ** 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: L0 2/0 99.90 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Sum 2/0 99.90 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-26T11:52:12.664 INFO:journalctl@ceph.mon.a.smithi156.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: ** Compaction Stats [default] ** 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 157.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-26T11:52:12.665 INFO:journalctl@ceph.mon.a.smithi156.stdout: Cumulative compaction: 0.00 GB write, 31.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: Interval compaction: 0.00 GB write, 31.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: Block cache BinnedLRUCache@0x55d17dc711f0#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: starting mon.a rank 0 at public addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] at bind addrs [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???) e1 preinit fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).mds e1 new map 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).mds e1 print_map 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: e1 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-26T11:52:12.666 INFO:journalctl@ceph.mon.a.smithi156.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout: legacy client fscid: -1 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout: 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout: No filesystems configured 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 bash[32006]: 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 systemd[1]: Started Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: monmap epoch 1 2024-05-26T11:52:12.667 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: last_changed 2024-05-26T11:52:04.693642+0000 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: min_mon_release 19 (squid) 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: election_strategy: 1 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: fsmap 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: osdmap e1: 0 total, 0 up, 0 in 2024-05-26T11:52:12.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:12 smithi156 ceph-mon[32025]: mgrmap e1: no daemons active 2024-05-26T11:52:13.546 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.904+0000 7f033d5a6700 1 Processor -- start 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.904+0000 7f033d5a6700 1 -- start start 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.904+0000 7f033d5a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f033d5a6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f033807d6b0 con 0x7f0338109a10 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f0336ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f0336ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58732/0 (socket says 172.21.15.156:58732) 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f0336ffd700 1 -- 172.21.15.156:0/1560490252 learned_addr learned my addr 172.21.15.156:0/1560490252 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f0336ffd700 1 -- 172.21.15.156:0/1560490252 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033810c340 con 0x7f0338109a10 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.905+0000 7f0336ffd700 1 --2- 172.21.15.156:0/1560490252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f0328009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ffbd1437ae2673f server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:13.547 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f03367fc700 1 -- 172.21.15.156:0/1560490252 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328004160 con 0x7f0338109a10 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f03367fc700 1 -- 172.21.15.156:0/1560490252 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f0328004300 con 0x7f0338109a10 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f03367fc700 1 -- 172.21.15.156:0/1560490252 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328011af0 con 0x7f0338109a10 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 -- 172.21.15.156:0/1560490252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 msgr2=0x7f033810be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 --2- 172.21.15.156:0/1560490252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f03280098a0 tx=0x7f0328009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 -- 172.21.15.156:0/1560490252 shutdown_connections 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 --2- 172.21.15.156:0/1560490252 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033810be00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 -- 172.21.15.156:0/1560490252 >> 172.21.15.156:0/1560490252 conn(0x7f0338103400 msgr2=0x7f0338105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 -- 172.21.15.156:0/1560490252 shutdown_connections 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.906+0000 7f033d5a6700 1 -- 172.21.15.156:0/1560490252 wait complete. 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f033d5a6700 1 Processor -- start 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f033d5a6700 1 -- start start 2024-05-26T11:52:13.548 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f033d5a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f033d5a6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f033810d2a0 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f0336ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f0336ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58748/0 (socket says 172.21.15.156:58748) 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.907+0000 7f0336ffd700 1 -- 172.21.15.156:0/2862617289 learned_addr learned my addr 172.21.15.156:0/2862617289 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0336ffd700 1 -- 172.21.15.156:0/2862617289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f033819d3c0 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0336ffd700 1 --2- 172.21.15.156:0/2862617289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0328009fb0 tx=0x7f0328011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f03280036a0 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f033819d650 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f032801a430 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0328019550 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03381a0340 con 0x7f0338109a10 2024-05-26T11:52:13.549 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0328019b90 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.908+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0328024730 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.909+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0338064ba0 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.911+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f0328043ec0 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.976+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f03381a0510 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.978+0000 7f0334ff9700 1 -- 172.21.15.156:0/2862617289 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f0328032030 con 0x7f0338109a10 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 msgr2=0x7f033819ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 --2- 172.21.15.156:0/2862617289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0328009fb0 tx=0x7f0328011940 comp rx=0 tx=0).stop 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 shutdown_connections 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 --2- 172.21.15.156:0/2862617289 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0338109a10 0x7f033819ce80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 >> 172.21.15.156:0/2862617289 conn(0x7f0338103400 msgr2=0x7f0338103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:13.550 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 shutdown_connections 2024-05-26T11:52:13.551 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:12.979+0000 7f033d5a6700 1 -- 172.21.15.156:0/2862617289 wait complete. 2024-05-26T11:52:13.551 INFO:teuthology.orchestra.run.smithi156.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-26T11:52:13.572 INFO:teuthology.orchestra.run.smithi156.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-26T11:52:13.573 INFO:teuthology.orchestra.run.smithi156.stdout:Creating mgr... 2024-05-26T11:52:13.573 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-26T11:52:13.573 INFO:teuthology.orchestra.run.smithi156.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-26T11:52:13.828 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x 2024-05-26T11:52:13.828 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Failed to reset failed state of unit ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x.service: Unit ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x.service not loaded. 2024-05-26T11:52:13.976 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968.target.wants/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@.service. 2024-05-26T11:52:14.279 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:13 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2862617289' entity='client.admin' 2024-05-26T11:52:14.467 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-26T11:52:14.467 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout disabled 2024-05-26T11:52:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-26T11:52:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:52:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:firewalld.service is not enabled 2024-05-26T11:52:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:Not possible to enable service . firewalld.service is not available 2024-05-26T11:52:14.483 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-26T11:52:14.483 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout disabled 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:firewalld.service is not enabled 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mgr to start... 2024-05-26T11:52:14.491 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mgr... 2024-05-26T11:52:15.248 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:15 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/274784732' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-26T11:52:15.248 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:15 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:15.160+0000 7f786a75b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T11:52:15.248 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:15 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:15.247+0000 7f786a75b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "health": { 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 0 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:15.645 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "a" 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-26T11:52:15.646 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-26T11:52:15.647 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "restful" 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:15.648 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modified": "2024-05-26T11:52:07.537745+0000", 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d407d5700 1 Processor -- start 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d407d5700 1 -- start start 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d407d5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d407d5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3807d660 con 0x7f4d38109a60 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d3e571700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d3e571700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42114/0 (socket says 172.21.15.156:42114) 2024-05-26T11:52:15.649 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.950+0000 7f4d3e571700 1 -- 172.21.15.156:0/4218742070 learned_addr learned my addr 172.21.15.156:0/4218742070 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.951+0000 7f4d3e571700 1 -- 172.21.15.156:0/4218742070 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3810c390 con 0x7f4d38109a60 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.951+0000 7f4d3e571700 1 --2- 172.21.15.156:0/4218742070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4d340098a0 tx=0x7f4d34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f610b7afb22573 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.951+0000 7f4d3d56f700 1 -- 172.21.15.156:0/4218742070 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d34004160 con 0x7f4d38109a60 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.951+0000 7f4d3d56f700 1 -- 172.21.15.156:0/4218742070 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d34004300 con 0x7f4d38109a60 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.951+0000 7f4d3d56f700 1 -- 172.21.15.156:0/4218742070 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d34011af0 con 0x7f4d38109a60 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 -- 172.21.15.156:0/4218742070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 msgr2=0x7f4d3810be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 --2- 172.21.15.156:0/4218742070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4d340098a0 tx=0x7f4d34009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 -- 172.21.15.156:0/4218742070 shutdown_connections 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 --2- 172.21.15.156:0/4218742070 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3810be50 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 -- 172.21.15.156:0/4218742070 >> 172.21.15.156:0/4218742070 conn(0x7f4d38103450 msgr2=0x7f4d38105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 -- 172.21.15.156:0/4218742070 shutdown_connections 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 -- 172.21.15.156:0/4218742070 wait complete. 2024-05-26T11:52:15.650 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.952+0000 7f4d407d5700 1 Processor -- start 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d407d5700 1 -- start start 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d407d5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d407d5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3810d290 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d3e571700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d3e571700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42124/0 (socket says 172.21.15.156:42124) 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d3e571700 1 -- 172.21.15.156:0/274784732 learned_addr learned my addr 172.21.15.156:0/274784732 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d3e571700 1 -- 172.21.15.156:0/274784732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3819d3d0 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d3e571700 1 --2- 172.21.15.156:0/274784732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009fb0 tx=0x7f4d34011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.953+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d340036a0 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3819d660 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4d3401a430 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4d34019550 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d381a0350 con 0x7f4d38109a60 2024-05-26T11:52:15.651 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4d34019b90 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.954+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4d34024730 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.955+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d38064ba0 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:14.959+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f4d34043ec0 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.032+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f4d381a0890 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.033+0000 7f4d2f7fe700 1 -- 172.21.15.156:0/274784732 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f4d34032030 con 0x7f4d38109a60 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 msgr2=0x7f4d3819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 --2- 172.21.15.156:0/274784732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009fb0 tx=0x7f4d34011940 comp rx=0 tx=0).stop 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 shutdown_connections 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 --2- 172.21.15.156:0/274784732 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4d38109a60 0x7f4d3819ce90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 >> 172.21.15.156:0/274784732 conn(0x7f4d38103450 msgr2=0x7f4d38103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 shutdown_connections 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:15.034+0000 7f4d407d5700 1 -- 172.21.15.156:0/274784732 wait complete. 2024-05-26T11:52:15.652 INFO:teuthology.orchestra.run.smithi156.stdout:mgr not available, waiting (1/15)... 2024-05-26T11:52:16.637 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:16.466+0000 7f786a75b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T11:52:16.637 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:16.636+0000 7f786a75b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T11:52:17.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:16.809+0000 7f786a75b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T11:52:17.393 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:17 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:17.097+0000 7f786a75b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T11:52:17.393 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:17 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:17.188+0000 7f786a75b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T11:52:17.393 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:17 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:17.392+0000 7f786a75b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T11:52:17.768 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:17 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:17.584+0000 7f786a75b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:52:18.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:18 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/934944986' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-26T11:52:18.295 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:18 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:18.217+0000 7f786a75b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T11:52:18.758 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:18 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:18.443+0000 7f786a75b200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "health": { 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.762 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 0 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "a" 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-26T11:52:18.763 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:18.764 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-26T11:52:18.764 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-26T11:52:18.771 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-26T11:52:18.772 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "restful" 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modified": "2024-05-26T11:52:07.537745+0000", 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:52:18.773 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd10de5700 1 Processor -- start 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd10de5700 1 -- start start 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd10de5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd10de5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c07ce50 con 0x7fcd0c107820 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd0a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.048+0000 7fcd0a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42140/0 (socket says 172.21.15.156:42140) 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0a59c700 1 -- 172.21.15.156:0/4140175787 learned_addr learned my addr 172.21.15.156:0/4140175787 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0a59c700 1 -- 172.21.15.156:0/4140175787 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0c108180 con 0x7fcd0c107820 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0a59c700 1 --2- 172.21.15.156:0/4140175787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd000098a0 tx=0x7fcd00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebaff7832c8f3eb server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0959a700 1 -- 172.21.15.156:0/4140175787 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd00004160 con 0x7fcd0c107820 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0959a700 1 -- 172.21.15.156:0/4140175787 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcd00004300 con 0x7fcd0c107820 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.049+0000 7fcd0959a700 1 -- 172.21.15.156:0/4140175787 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd00011af0 con 0x7fcd0c107820 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 -- 172.21.15.156:0/4140175787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 msgr2=0x7fcd0c107c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:18.774 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 --2- 172.21.15.156:0/4140175787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcd000098a0 tx=0x7fcd00009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 -- 172.21.15.156:0/4140175787 shutdown_connections 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 --2- 172.21.15.156:0/4140175787 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c107c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 -- 172.21.15.156:0/4140175787 >> 172.21.15.156:0/4140175787 conn(0x7fcd0c103420 msgr2=0x7fcd0c105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 -- 172.21.15.156:0/4140175787 shutdown_connections 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.050+0000 7fcd10de5700 1 -- 172.21.15.156:0/4140175787 wait complete. 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd10de5700 1 Processor -- start 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd10de5700 1 -- start start 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd10de5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd10de5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcd0c1091b0 con 0x7fcd0c107820 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd0a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd0a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42148/0 (socket says 172.21.15.156:42148) 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd0a59c700 1 -- 172.21.15.156:0/934944986 learned_addr learned my addr 172.21.15.156:0/934944986 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:18.775 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.051+0000 7fcd0a59c700 1 -- 172.21.15.156:0/934944986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd0c19f630 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fcd0a59c700 1 --2- 172.21.15.156:0/934944986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009fb0 tx=0x7fcd00011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd000036a0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd0c19f8c0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcd0001a430 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcd00019550 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.052+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd0c1a25b0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.053+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcd00019be0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.053+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcd00024730 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.053+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd0c19fda0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.056+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fcd00043ec0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.129+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcd0c064ba0 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.130+0000 7fccfb7fe700 1 -- 172.21.15.156:0/934944986 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fcd00032030 con 0x7fcd0c107820 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.130+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 msgr2=0x7fcd0c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:18.776 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.130+0000 7fcd10de5700 1 --2- 172.21.15.156:0/934944986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcd00009fb0 tx=0x7fcd00011940 comp rx=0 tx=0).stop 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.131+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 shutdown_connections 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.131+0000 7fcd10de5700 1 --2- 172.21.15.156:0/934944986 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcd0c107820 0x7fcd0c19f0f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.131+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 >> 172.21.15.156:0/934944986 conn(0x7fcd0c103420 msgr2=0x7fcd0c195cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.131+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 shutdown_connections 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:18.131+0000 7fcd10de5700 1 -- 172.21.15.156:0/934944986 wait complete. 2024-05-26T11:52:18.777 INFO:teuthology.orchestra.run.smithi156.stdout:mgr not available, waiting (2/15)... 2024-05-26T11:52:19.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:18 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:18.875+0000 7f786a75b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T11:52:19.045 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:18 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:18.972+0000 7f786a75b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T11:52:19.699 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:19 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:19.344+0000 7f786a75b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T11:52:19.699 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:19 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:19.698+0000 7f786a75b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T11:52:20.040 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:19 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:19.791+0000 7f786a75b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T11:52:20.294 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:20 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:20.040+0000 7f786a75b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T11:52:20.295 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:20 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:20.177+0000 7f786a75b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T11:52:20.295 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:20 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:20.274+0000 7f786a75b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T11:52:20.889 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:20 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:20.608+0000 7f786a75b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T11:52:20.889 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:20 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:20.701+0000 7f786a75b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T11:52:21.590 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:21 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:21.317+0000 7f786a75b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T11:52:21.590 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:21 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:21.537+0000 7f786a75b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T11:52:22.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: Activating manager daemon x 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: mgrmap e2: x(active, starting, since 0.00329182s) 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: Manager daemon x is now available 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' 2024-05-26T11:52:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:21 smithi156 ceph-mon[32025]: from='mgr.14100 172.21.15.156:0/4213274372' entity='mgr.x' 2024-05-26T11:52:23.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:23 smithi156 ceph-mon[32025]: mgrmap e3: x(active, since 1.00805s) 2024-05-26T11:52:24.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:24 smithi156 ceph-mon[32025]: mgrmap e4: x(active, since 2s) 2024-05-26T11:52:24.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:24 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/585208988' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "health": { 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 0 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "a" 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-26T11:52:30.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:30.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-26T11:52:30.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-26T11:52:30.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-26T11:52:30.988 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-26T11:52:30.988 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.989 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "restful" 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ], 2024-05-26T11:52:30.990 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "modified": "2024-05-26T11:52:07.537745+0000", 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout }, 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.147+0000 7ff6326ac700 1 Processor -- start 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff6326ac700 1 -- start start 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff6326ac700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff6326ac700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c07d6a0 con 0x7ff62c107810 2024-05-26T11:52:30.991 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42224/0 (socket says 172.21.15.156:42224) 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62bfff700 1 -- 172.21.15.156:0/532164290 learned_addr learned my addr 172.21.15.156:0/532164290 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62bfff700 1 -- 172.21.15.156:0/532164290 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c108170 con 0x7ff62c107810 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62bfff700 1 --2- 172.21.15.156:0/532164290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0098a0 tx=0x7ff61c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc12db7d43caef3 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.148+0000 7ff62b7fe700 1 -- 172.21.15.156:0/532164290 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff61c004160 con 0x7ff62c107810 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff62b7fe700 1 -- 172.21.15.156:0/532164290 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff61c004300 con 0x7ff62c107810 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff62b7fe700 1 -- 172.21.15.156:0/532164290 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff61c011af0 con 0x7ff62c107810 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 -- 172.21.15.156:0/532164290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 msgr2=0x7ff62c107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 --2- 172.21.15.156:0/532164290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7ff61c0098a0 tx=0x7ff61c009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 -- 172.21.15.156:0/532164290 shutdown_connections 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 --2- 172.21.15.156:0/532164290 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c107c30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 -- 172.21.15.156:0/532164290 >> 172.21.15.156:0/532164290 conn(0x7ff62c103450 msgr2=0x7ff62c105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 -- 172.21.15.156:0/532164290 shutdown_connections 2024-05-26T11:52:30.992 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.149+0000 7ff6326ac700 1 -- 172.21.15.156:0/532164290 wait complete. 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff6326ac700 1 Processor -- start 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff6326ac700 1 -- start start 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff6326ac700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff6326ac700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff62c1091a0 con 0x7ff62c107810 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff62bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff62bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42228/0 (socket says 172.21.15.156:42228) 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff62bfff700 1 -- 172.21.15.156:0/585208988 learned_addr learned my addr 172.21.15.156:0/585208988 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff62bfff700 1 -- 172.21.15.156:0/585208988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff62c19f5f0 con 0x7ff62c107810 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.150+0000 7ff62bfff700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009fb0 tx=0x7ff61c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.151+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff61c0036a0 con 0x7ff62c107810 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.151+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff61c01a430 con 0x7ff62c107810 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.151+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff62c19f880 con 0x7ff62c107810 2024-05-26T11:52:30.993 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.151+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff61c019550 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.151+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff62c1a2570 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7ff61c003840 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff62c064ba0 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff629ffb700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 0x7ff618040890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff61c053830 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff623fff700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 0x7ff618040890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.152+0000 7ff623fff700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 0x7ff618040890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff614006fd0 tx=0x7ff614006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.157+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff61c00b3e0 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.460+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7ff62c0008d0 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.460+0000 7ff629ffb700 1 -- 172.21.15.156:0/585208988 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7ff61c021e80 con 0x7ff62c107810 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 msgr2=0x7ff618040890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 0x7ff618040890 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff614006fd0 tx=0x7ff614006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:30.994 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 msgr2=0x7ff62c19f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7ff61c009fb0 tx=0x7ff61c011940 comp rx=0 tx=0).stop 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 shutdown_connections 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff61803e3d0 0x7ff618040890 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 --2- 172.21.15.156:0/585208988 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff62c107810 0x7ff62c19f0b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 >> 172.21.15.156:0/585208988 conn(0x7ff62c103450 msgr2=0x7ff62c195cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 shutdown_connections 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:24.463+0000 7ff6326ac700 1 -- 172.21.15.156:0/585208988 wait complete. 2024-05-26T11:52:30.995 INFO:teuthology.orchestra.run.smithi156.stdout:mgr is available 2024-05-26T11:52:32.435 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:32 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/848371990' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [global] 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout fsid = 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:32.688 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout [osd] 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbe1cb9700 1 Processor -- start 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbe1cb9700 1 -- start start 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbe1cb9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbe1cb9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc1087c0 con 0x7fcbdc107e60 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbdb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:32.689 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbdb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52768/0 (socket says 172.21.15.156:52768) 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.866+0000 7fcbdb7fe700 1 -- 172.21.15.156:0/1656050381 learned_addr learned my addr 172.21.15.156:0/1656050381 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.867+0000 7fcbdb7fe700 1 -- 172.21.15.156:0/1656050381 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc108fc0 con 0x7fcbdc107e60 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.867+0000 7fcbdb7fe700 1 --2- 172.21.15.156:0/1656050381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0098a0 tx=0x7fcbcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec6174dc3310e5c server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.867+0000 7fcbda7fc700 1 -- 172.21.15.156:0/1656050381 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbcc004160 con 0x7fcbdc107e60 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.867+0000 7fcbda7fc700 1 -- 172.21.15.156:0/1656050381 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcbcc004300 con 0x7fcbdc107e60 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.867+0000 7fcbda7fc700 1 -- 172.21.15.156:0/1656050381 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbcc011af0 con 0x7fcbdc107e60 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/1656050381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 msgr2=0x7fcbdc108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/1656050381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc0098a0 tx=0x7fcbcc009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/1656050381 shutdown_connections 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/1656050381 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc108280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/1656050381 >> 172.21.15.156:0/1656050381 conn(0x7fcbdc103480 msgr2=0x7fcbdc1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:32.690 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/1656050381 shutdown_connections 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/1656050381 wait complete. 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 Processor -- start 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.868+0000 7fcbe1cb9700 1 -- start start 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbe1cb9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbe1cb9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcbdc10c400 con 0x7fcbdc107e60 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbdb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbdb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52776/0 (socket says 172.21.15.156:52776) 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbdb7fe700 1 -- 172.21.15.156:0/848371990 learned_addr learned my addr 172.21.15.156:0/848371990 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbdb7fe700 1 -- 172.21.15.156:0/848371990 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbdc19d430 con 0x7fcbdc107e60 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbdb7fe700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009fb0 tx=0x7fcbcc011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbcc0036a0 con 0x7fcbdc107e60 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fcbcc01a430 con 0x7fcbdc107e60 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbdc19d6c0 con 0x7fcbdc107e60 2024-05-26T11:52:32.691 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbcc019550 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.869+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbdc1a03b0 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.870+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7fcbcc019850 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.870+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbdc064ba0 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.870+0000 7fcbd8ff9700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 0x7fcbc4040890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.871+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcbcc0539f0 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.871+0000 7fcbdaffd700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 0x7fcbc4040890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.871+0000 7fcbdaffd700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 0x7fcbc4040890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006fd0 tx=0x7fcbd0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:31.876+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcbcc053410 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.074+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fcbdc0008d0 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.075+0000 7fcbd8ff9700 1 -- 172.21.15.156:0/848371990 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+409 (secure 0 0 0) 0x7fcbcc01a5e0 con 0x7fcbdc107e60 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 msgr2=0x7fcbc4040890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 0x7fcbc4040890 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcbd0006fd0 tx=0x7fcbd0006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 msgr2=0x7fcbdc19cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc009fb0 tx=0x7fcbcc011940 comp rx=0 tx=0).stop 2024-05-26T11:52:32.692 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 shutdown_connections 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fcbc403e3d0 0x7fcbc4040890 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 --2- 172.21.15.156:0/848371990 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcbdc107e60 0x7fcbdc19cef0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.077+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 >> 172.21.15.156:0/848371990 conn(0x7fcbdc103480 msgr2=0x7fcbdc104300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.078+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 shutdown_connections 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:32.078+0000 7fcbe1cb9700 1 -- 172.21.15.156:0/848371990 wait complete. 2024-05-26T11:52:32.693 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling cephadm module... 2024-05-26T11:52:33.691 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:33 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3688534526' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-26T11:52:33.692 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: ignoring --setuser ceph since I am not root 2024-05-26T11:52:33.692 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: ignoring --setgroup ceph since I am not root 2024-05-26T11:52:33.692 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:33.547+0000 7f46f82d0200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T11:52:33.967 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:33.734+0000 7f46f82d0200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:52:33.967 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:33.838+0000 7f46f82d0200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T11:52:33.969 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.058+0000 7f74e1466700 1 Processor -- start 2024-05-26T11:52:33.978 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.058+0000 7f74e1466700 1 -- start start 2024-05-26T11:52:33.978 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.058+0000 7f74e1466700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.058+0000 7f74e1466700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc07d740 con 0x7f74dc109980 2024-05-26T11:52:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.058+0000 7f74daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52782/0 (socket says 172.21.15.156:52782) 2024-05-26T11:52:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74daffd700 1 -- 172.21.15.156:0/1216567002 learned_addr learned my addr 172.21.15.156:0/1216567002 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:33.979 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74daffd700 1 -- 172.21.15.156:0/1216567002 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc10c2b0 con 0x7f74dc109980 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74daffd700 1 --2- 172.21.15.156:0/1216567002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098a0 tx=0x7f74cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b32c59e37ff485 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74da7fc700 1 -- 172.21.15.156:0/1216567002 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc004160 con 0x7f74dc109980 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74da7fc700 1 -- 172.21.15.156:0/1216567002 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f74cc004300 con 0x7f74dc109980 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.059+0000 7f74da7fc700 1 -- 172.21.15.156:0/1216567002 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc011af0 con 0x7f74dc109980 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 -- 172.21.15.156:0/1216567002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 msgr2=0x7f74dc10bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 --2- 172.21.15.156:0/1216567002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f74cc0098a0 tx=0x7f74cc009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:33.980 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 -- 172.21.15.156:0/1216567002 shutdown_connections 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 --2- 172.21.15.156:0/1216567002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc10bd70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 -- 172.21.15.156:0/1216567002 >> 172.21.15.156:0/1216567002 conn(0x7f74dc103450 msgr2=0x7f74dc105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 -- 172.21.15.156:0/1216567002 shutdown_connections 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 -- 172.21.15.156:0/1216567002 wait complete. 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.060+0000 7f74e1466700 1 Processor -- start 2024-05-26T11:52:33.981 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74e1466700 1 -- start start 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74e1466700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74e1466700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f74dc10d280 con 0x7f74dc109980 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52784/0 (socket says 172.21.15.156:52784) 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74daffd700 1 -- 172.21.15.156:0/3688534526 learned_addr learned my addr 172.21.15.156:0/3688534526 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74daffd700 1 -- 172.21.15.156:0/3688534526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74dc19d410 con 0x7f74dc109980 2024-05-26T11:52:33.982 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74daffd700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f74cc011ac0 tx=0x7f74cc011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc003970 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74dc19d6a0 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f74cc003b10 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74cc024610 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.061+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74dc1a0390 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.062+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 4) ==== 50211+0+0 (secure 0 0 0) 0x7f74cc024890 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.062+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74dc064ba0 con 0x7f74dc109980 2024-05-26T11:52:33.983 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.062+0000 7f74d8ff9700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 0x7f74c80408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.063+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f74cc053a10 con 0x7f74dc109980 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.063+0000 7f74d27fc700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 0x7f74c80408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.063+0000 7f74d27fc700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 0x7f74c80408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.068+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f74cc02c350 con 0x7f74dc109980 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.326+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f74dc1a0560 con 0x7f74dc109980 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.377+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f74dc1a0560 con 0x7f74dc109980 2024-05-26T11:52:33.984 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.379+0000 7f74d8ff9700 1 -- 172.21.15.156:0/3688534526 <== mon.0 v2:172.21.15.156:3300/0 8 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7f74cc02d3f0 con 0x7f74dc109980 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 msgr2=0x7f74c80408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 0x7f74c80408e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f74c4006fd0 tx=0x7f74c4006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 msgr2=0x7f74dc19ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f74cc011ac0 tx=0x7f74cc011ba0 comp rx=0 tx=0).stop 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 shutdown_connections 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7f74c803e420 0x7f74c80408e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:33.985 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 --2- 172.21.15.156:0/3688534526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f74dc109980 0x7f74dc19ced0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 >> 172.21.15.156:0/3688534526 conn(0x7f74dc103450 msgr2=0x7f74dc104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.380+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 shutdown_connections 2024-05-26T11:52:33.986 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:33.381+0000 7f74e1466700 1 -- 172.21.15.156:0/3688534526 wait complete. 2024-05-26T11:52:34.245 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:34.155+0000 7f46f82d0200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T11:52:34.245 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:34.244+0000 7f46f82d0200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T11:52:34.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:34 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3688534526' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-26T11:52:34.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:34 smithi156 ceph-mon[32025]: mgrmap e5: x(active, since 11s) 2024-05-26T11:52:35.339 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.387+0000 7fa0e3d3f700 1 Processor -- start 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.387+0000 7fa0e3d3f700 1 -- start start 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.387+0000 7fa0e3d3f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:35.340 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.387+0000 7fa0e3d3f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc1098d0 con 0x7fa0dc108f70 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.387+0000 7fa0e1adb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e1adb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52816/0 (socket says 172.21.15.156:52816) 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e1adb700 1 -- 172.21.15.156:0/2179919415 learned_addr learned my addr 172.21.15.156:0/2179919415 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e1adb700 1 -- 172.21.15.156:0/2179919415 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc10a080 con 0x7fa0dc108f70 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e1adb700 1 --2- 172.21.15.156:0/2179919415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80098a0 tx=0x7fa0d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b140ec3b75c0517 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e0ad9700 1 -- 172.21.15.156:0/2179919415 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d8004160 con 0x7fa0dc108f70 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e0ad9700 1 -- 172.21.15.156:0/2179919415 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa0d8004300 con 0x7fa0dc108f70 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.388+0000 7fa0e0ad9700 1 -- 172.21.15.156:0/2179919415 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d8011af0 con 0x7fa0dc108f70 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/2179919415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 msgr2=0x7fa0dc109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/2179919415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa0d80098a0 tx=0x7fa0d8009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/2179919415 shutdown_connections 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/2179919415 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc109390 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.341 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/2179919415 >> 172.21.15.156:0/2179919415 conn(0x7fa0dc104530 msgr2=0x7fa0dc106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/2179919415 shutdown_connections 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.389+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/2179919415 wait complete. 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e3d3f700 1 Processor -- start 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e3d3f700 1 -- start start 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e3d3f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e3d3f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa0dc10d4c0 con 0x7fa0dc108f70 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e1adb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e1adb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:52822/0 (socket says 172.21.15.156:52822) 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e1adb700 1 -- 172.21.15.156:0/3427820962 learned_addr learned my addr 172.21.15.156:0/3427820962 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e1adb700 1 -- 172.21.15.156:0/3427820962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0dc19d3f0 con 0x7fa0dc108f70 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.390+0000 7fa0e1adb700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8011ac0 tx=0x7fa0d8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:35.342 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.391+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d8003970 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.391+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa0d8003b10 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.391+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0dc19d680 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.391+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0dc1a0370 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.391+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa0d8024610 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0c0005360 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7fa0d8024860 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0d2ffd700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 0x7fa0c8040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa0d80537e0 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0e12da700 1 -- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 msgr2=0x7fa0c8040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.392+0000 7fa0e12da700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 0x7fa0c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.397+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa0d8053200 con 0x7fa0dc108f70 2024-05-26T11:52:35.343 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.593+0000 7fa0e12da700 1 -- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 msgr2=0x7fa0c8040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.593+0000 7fa0e12da700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 0x7fa0c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.702+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7fa0c0005e60 con 0x7fa0dc108f70 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.703+0000 7fa0d2ffd700 1 -- 172.21.15.156:0/3427820962 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7fa0d802d3f0 con 0x7fa0dc108f70 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 msgr2=0x7fa0c8040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 0x7fa0c8040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 msgr2=0x7fa0dc19ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa0d8011ac0 tx=0x7fa0d8011ba0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 shutdown_connections 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7fa0c803e4c0 0x7fa0c8040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 --2- 172.21.15.156:0/3427820962 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa0dc108f70 0x7fa0dc19ceb0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.705+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 >> 172.21.15.156:0/3427820962 conn(0x7fa0dc104530 msgr2=0x7fa0dc105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.706+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 shutdown_connections 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:34.706+0000 7fa0e3d3f700 1 -- 172.21.15.156:0/3427820962 wait complete. 2024-05-26T11:52:35.344 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for the mgr to restart... 2024-05-26T11:52:35.345 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mgr epoch 5... 2024-05-26T11:52:35.641 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:35 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3427820962' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-26T11:52:35.641 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:35.467+0000 7f46f82d0200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T11:52:35.641 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:35.640+0000 7f46f82d0200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T11:52:35.909 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:35.835+0000 7f46f82d0200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T11:52:36.223 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:36.131+0000 7f46f82d0200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T11:52:36.518 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:36.223+0000 7f46f82d0200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T11:52:36.518 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:36.428+0000 7f46f82d0200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T11:52:36.794 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:36.621+0000 7f46f82d0200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:52:37.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:37.247+0000 7f46f82d0200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T11:52:37.545 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:37.468+0000 7f46f82d0200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-26T11:52:38.294 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:37.894+0000 7f46f82d0200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T11:52:38.295 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:37.991+0000 7f46f82d0200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T11:52:38.709 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:38.370+0000 7f46f82d0200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T11:52:39.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:38.740+0000 7f46f82d0200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T11:52:39.045 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:38.835+0000 7f46f82d0200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T11:52:39.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:39.080+0000 7f46f82d0200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T11:52:39.545 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:39.216+0000 7f46f82d0200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T11:52:39.545 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:39.319+0000 7f46f82d0200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T11:52:40.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:39.646+0000 7f46f82d0200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T11:52:40.045 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:39.740+0000 7f46f82d0200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T11:52:40.614 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:40.342+0000 7f46f82d0200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T11:52:40.614 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:40.561+0000 7f46f82d0200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: Active manager daemon x restarted 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: Activating manager daemon x 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: osdmap e2: 0 total, 0 up, 0 in 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: mgrmap e6: x(active, starting, since 0.00361259s) 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:52:41.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: Manager daemon x is now available 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:52:41.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-26T11:52:41.047 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:40 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-26T11:52:41.887 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:41 smithi156 ceph-mon[32025]: Found migration_current of "None". Setting to last migration. 2024-05-26T11:52:41.887 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:41 smithi156 ceph-mon[32025]: mgrmap e7: x(active, since 1.00589s) 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.908+0000 7ff84172c700 1 Processor -- start 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.908+0000 7ff84172c700 1 -- start start 2024-05-26T11:52:42.147 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.908+0000 7ff84172c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.908+0000 7ff84172c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c1087c0 con 0x7ff83c107e60 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff83affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff83affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56152/0 (socket says 172.21.15.156:56152) 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff83affd700 1 -- 172.21.15.156:0/1405640925 learned_addr learned my addr 172.21.15.156:0/1405640925 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff83affd700 1 -- 172.21.15.156:0/1405640925 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c108f70 con 0x7ff83c107e60 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff83affd700 1 --2- 172.21.15.156:0/1405640925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8300098a0 tx=0x7ff830009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb18389959faac28 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff839ffb700 1 -- 172.21.15.156:0/1405640925 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff830004160 con 0x7ff83c107e60 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff839ffb700 1 -- 172.21.15.156:0/1405640925 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff830004300 con 0x7ff83c107e60 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.909+0000 7ff839ffb700 1 -- 172.21.15.156:0/1405640925 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff830011af0 con 0x7ff83c107e60 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 -- 172.21.15.156:0/1405640925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 msgr2=0x7ff83c108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 --2- 172.21.15.156:0/1405640925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff8300098a0 tx=0x7ff830009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.148 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 -- 172.21.15.156:0/1405640925 shutdown_connections 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 --2- 172.21.15.156:0/1405640925 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c108280 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 -- 172.21.15.156:0/1405640925 >> 172.21.15.156:0/1405640925 conn(0x7ff83c103400 msgr2=0x7ff83c105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 -- 172.21.15.156:0/1405640925 shutdown_connections 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.910+0000 7ff84172c700 1 -- 172.21.15.156:0/1405640925 wait complete. 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff84172c700 1 Processor -- start 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff84172c700 1 -- start start 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff84172c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff84172c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff83c07c860 con 0x7ff83c107e60 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff83affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff83affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56164/0 (socket says 172.21.15.156:56164) 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.911+0000 7ff83affd700 1 -- 172.21.15.156:0/1484060895 learned_addr learned my addr 172.21.15.156:0/1484060895 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff83affd700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c19d400 con 0x7ff83c107e60 2024-05-26T11:52:42.149 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff83affd700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fb0 tx=0x7ff830011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff8300036a0 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7ff83001a430 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff83c19d690 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff830019550 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.912+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff83c1a0380 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 5) ==== 50222+0+0 (secure 0 0 0) 0x7ff830019850 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff823fff700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff824040ff0 con 0x7ff82403e420 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff8300538d0 con 0x7ff83c107e60 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff83a7fc700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:35.913+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:36.114+0000 7ff83a7fc700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:42.150 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:36.114+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:36.515+0000 7ff83a7fc700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:36.515+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:37.316+0000 7ff83a7fc700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:37.316+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:38.917+0000 7ff83a7fc700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2544075824 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:38.917+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:40.564+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mgrmap(e 6) ==== 49989+0+0 (secure 0 0 0) 0x7ff830019b90 con 0x7ff83c107e60 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:40.564+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 msgr2=0x7ff8240408e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:40.564+0000 7ff823fff700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.565+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7ff8300558e0 con 0x7ff83c107e60 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.566+0000 7ff823fff700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 0x7ff824044100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.566+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff824040ff0 con 0x7ff824041d10 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.566+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 0x7ff824044100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:42.151 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.566+0000 7ff83a7fc700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 0x7ff824044100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff82c003a10 tx=0x7ff82c009290 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.567+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7ff824040ff0 con 0x7ff824041d10 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.575+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff83c06a680 con 0x7ff824041d10 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff823fff700 1 -- 172.21.15.156:0/1484060895 <== mgr.14118 v2:172.21.15.156:6800/2588638081 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff83c06a680 con 0x7ff824041d10 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 msgr2=0x7ff824044100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 0x7ff824044100 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff82c003a10 tx=0x7ff82c009290 comp rx=0 tx=0).stop 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 msgr2=0x7ff83c19cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff830009fb0 tx=0x7ff830011940 comp rx=0 tx=0).stop 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 shutdown_connections 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff83c107e60 0x7ff83c19cec0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7ff824041d10 0x7ff824044100 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 --2- 172.21.15.156:0/1484060895 >> [v2:172.21.15.156:6800/2544075824,v1:172.21.15.156:6801/2544075824] conn(0x7ff82403e420 0x7ff8240408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 >> 172.21.15.156:0/1484060895 conn(0x7ff83c103400 msgr2=0x7ff83c1040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 shutdown_connections 2024-05-26T11:52:42.152 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:41.576+0000 7ff84172c700 1 -- 172.21.15.156:0/1484060895 wait complete. 2024-05-26T11:52:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:mgr epoch 5 is available 2024-05-26T11:52:42.153 INFO:teuthology.orchestra.run.smithi156.stdout:Setting orchestrator backend to cephadm... 2024-05-26T11:52:43.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:42 smithi156 ceph-mon[32025]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-26T11:52:43.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:42 smithi156 ceph-mon[32025]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-26T11:52:43.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:42 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:43.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:42 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:43.368 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe3226ae700 1 Processor -- start 2024-05-26T11:52:43.368 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe3226ae700 1 -- start start 2024-05-26T11:52:43.368 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe3226ae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:43.368 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe3226ae700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c10c0a0 con 0x7fe31c07d580 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56248/0 (socket says 172.21.15.156:56248) 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe31bfff700 1 -- 172.21.15.156:0/3810903359 learned_addr learned my addr 172.21.15.156:0/3810903359 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe31bfff700 1 -- 172.21.15.156:0/3810903359 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c10c220 con 0x7fe31c07d580 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.549+0000 7fe31bfff700 1 --2- 172.21.15.156:0/3810903359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0098a0 tx=0x7fe30c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a50c1c7ce816a7f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe31affd700 1 -- 172.21.15.156:0/3810903359 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c004160 con 0x7fe31c07d580 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe31affd700 1 -- 172.21.15.156:0/3810903359 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe30c004300 con 0x7fe31c07d580 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe31affd700 1 -- 172.21.15.156:0/3810903359 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c011af0 con 0x7fe31c07d580 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe3226ae700 1 -- 172.21.15.156:0/3810903359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 msgr2=0x7fe31c10bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:43.369 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe3226ae700 1 --2- 172.21.15.156:0/3810903359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0098a0 tx=0x7fe30c009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe3226ae700 1 -- 172.21.15.156:0/3810903359 shutdown_connections 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe3226ae700 1 --2- 172.21.15.156:0/3810903359 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c07d580 0x7fe31c10bad0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.550+0000 7fe3226ae700 1 -- 172.21.15.156:0/3810903359 >> 172.21.15.156:0/3810903359 conn(0x7fe31c103430 msgr2=0x7fe31c105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 -- 172.21.15.156:0/3810903359 shutdown_connections 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 -- 172.21.15.156:0/3810903359 wait complete. 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 Processor -- start 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 -- start start 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.551+0000 7fe3226ae700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c10d2a0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe31bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56262/0 (socket says 172.21.15.156:56262) 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe31bfff700 1 -- 172.21.15.156:0/256991174 learned_addr learned my addr 172.21.15.156:0/256991174 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:43.370 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe31bfff700 1 -- 172.21.15.156:0/256991174 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c19d810 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe31bfff700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009fb0 tx=0x7fe30c011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c0036a0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe30c01a430 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c19daa0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe30c019550 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.552+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1a0580 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.553+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 7) ==== 50116+0+0 (secure 0 0 0) 0x7fe30c019850 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.553+0000 7fe3197fa700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 0x7fe3040407b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.553+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe30c0536b0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.553+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c064ba0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.554+0000 7fe31b7fe700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 0x7fe3040407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.554+0000 7fe31b7fe700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 0x7fe3040407b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe310006fd0 tx=0x7fe310006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:43.371 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.559+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe30c0243f0 con 0x7fe31c19ceb0 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.732+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fe30c02d440 con 0x7fe31c19ceb0 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.788+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fe31c1046a0 con 0x7fe30403e2f0 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.793+0000 7fe3197fa700 1 -- 172.21.15.156:0/256991174 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fe31c1046a0 con 0x7fe30403e2f0 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 msgr2=0x7fe3040407b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 0x7fe3040407b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe310006fd0 tx=0x7fe310006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 msgr2=0x7fe31c19d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe30c009fb0 tx=0x7fe30c011940 comp rx=0 tx=0).stop 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 shutdown_connections 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe30403e2f0 0x7fe3040407b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 --2- 172.21.15.156:0/256991174 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe31c19ceb0 0x7fe31c19d2d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.796+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 >> 172.21.15.156:0/256991174 conn(0x7fe31c103430 msgr2=0x7fe31c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.797+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 shutdown_connections 2024-05-26T11:52:43.372 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:42.797+0000 7fe3226ae700 1 -- 172.21.15.156:0/256991174 wait complete. 2024-05-26T11:52:44.018 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: mgrmap e8: x(active, since 2s) 2024-05-26T11:52:44.018 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:44.018 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:44.018 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:52:44.019 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: [26/May/2024:11:52:43] ENGINE Bus STARTING 2024-05-26T11:52:44.019 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: [26/May/2024:11:52:43] ENGINE Serving on https://172.21.15.156:7150 2024-05-26T11:52:44.019 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:43 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.797+0000 7f4aa9965700 1 Processor -- start 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.797+0000 7f4aa9965700 1 -- start start 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.797+0000 7f4aa9965700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.797+0000 7f4aa9965700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa4107c50 con 0x7f4aa4107260 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56272/0 (socket says 172.21.15.156:56272) 2024-05-26T11:52:44.658 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa2ffd700 1 -- 172.21.15.156:0/3976612721 learned_addr learned my addr 172.21.15.156:0/3976612721 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa2ffd700 1 -- 172.21.15.156:0/3976612721 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa4107dd0 con 0x7f4aa4107260 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa2ffd700 1 --2- 172.21.15.156:0/3976612721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a940098a0 tx=0x7f4a94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7914b4d29885c0ca server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa1ffb700 1 -- 172.21.15.156:0/3976612721 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94004160 con 0x7f4aa4107260 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa1ffb700 1 -- 172.21.15.156:0/3976612721 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a94004300 con 0x7f4aa4107260 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.798+0000 7f4aa1ffb700 1 -- 172.21.15.156:0/3976612721 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94011af0 con 0x7f4aa4107260 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 -- 172.21.15.156:0/3976612721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 msgr2=0x7f4aa4107680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 --2- 172.21.15.156:0/3976612721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4a940098a0 tx=0x7f4a94009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 -- 172.21.15.156:0/3976612721 shutdown_connections 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 --2- 172.21.15.156:0/3976612721 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4107680 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 -- 172.21.15.156:0/3976612721 >> 172.21.15.156:0/3976612721 conn(0x7f4aa4102be0 msgr2=0x7f4aa4105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 -- 172.21.15.156:0/3976612721 shutdown_connections 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 -- 172.21.15.156:0/3976612721 wait complete. 2024-05-26T11:52:44.659 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.799+0000 7f4aa9965700 1 Processor -- start 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa9965700 1 -- start start 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa9965700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa9965700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa410b0c0 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa2ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56274/0 (socket says 172.21.15.156:56274) 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa2ffd700 1 -- 172.21.15.156:0/4261497116 learned_addr learned my addr 172.21.15.156:0/4261497116 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa2ffd700 1 -- 172.21.15.156:0/4261497116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa410a0a0 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa2ffd700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a940117b0 tx=0x7f4a94011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94003af0 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa4108290 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.800+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4a94020070 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.801+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a94019630 con 0x7f4aa4107260 2024-05-26T11:52:44.660 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.801+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa41087d0 con 0x7f4aa4107260 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f4a940198f0 con 0x7f4aa4107260 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa406a680 con 0x7f4aa4107260 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa8963700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 0x7f4a8c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4a94053bd0 con 0x7f4aa4107260 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa27fc700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 0x7f4a8c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.802+0000 7f4aa27fc700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 0x7f4a8c040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006fd0 tx=0x7f4a98006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:43.807+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a940042a0 con 0x7f4aa4107260 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.017+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f4aa4103ea0 con 0x7f4a8c03e470 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.018+0000 7f4aa8963700 1 -- 172.21.15.156:0/4261497116 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f4aa4103ea0 con 0x7f4a8c03e470 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 msgr2=0x7f4a8c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 0x7f4a8c040930 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4a98006fd0 tx=0x7f4a98006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 msgr2=0x7f4aa4109b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4a940117b0 tx=0x7f4a94011b70 comp rx=0 tx=0).stop 2024-05-26T11:52:44.661 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 shutdown_connections 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4a8c03e470 0x7f4a8c040930 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 --2- 172.21.15.156:0/4261497116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4aa4107260 0x7f4aa4109b60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 >> 172.21.15.156:0/4261497116 conn(0x7f4aa4102be0 msgr2=0x7f4aa4103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 shutdown_connections 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:44.021+0000 7f4aa9965700 1 -- 172.21.15.156:0/4261497116 wait complete. 2024-05-26T11:52:44.662 INFO:teuthology.orchestra.run.smithi156.stdout:Generating ssh key... 2024-05-26T11:52:44.987 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:44 smithi156 ceph-mon[32025]: [26/May/2024:11:52:43] ENGINE Serving on http://172.21.15.156:8765 2024-05-26T11:52:44.987 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:44 smithi156 ceph-mon[32025]: [26/May/2024:11:52:43] ENGINE Bus STARTED 2024-05-26T11:52:44.987 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:44 smithi156 ceph-mon[32025]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:45.923 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:45 smithi156 ceph-mon[32025]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:45 smithi156 ceph-mon[32025]: Generating ssh key... 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:45 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:45 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: Generating public/private rsa key pair. 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: Your identification has been saved in /tmp/tmptofvl188/key. 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: Your public key has been saved in /tmp/tmptofvl188/key.pub. 2024-05-26T11:52:45.924 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: The key fingerprint is: 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: SHA256:PQxgtAPVlKN6+pqDltxDO/3fnYweMdFXjQe3Tv1lBv0 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: The key's randomart image is: 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: +---[RSA 3072]----+ 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | .o=o.. o++| 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | o o+ ..oB| 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | o... . .+B| 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | .. + .o+E| 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | . S + o ..| 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | o . . o | 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | . = = . | 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | = B.. . = . | 2024-05-26T11:52:45.925 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: | . o*......+ + | 2024-05-26T11:52:45.926 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: +----[SHA256]-----+ 2024-05-26T11:52:46.214 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.085+0000 7f4bf84ab700 1 Processor -- start 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.085+0000 7f4bf84ab700 1 -- start start 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf84ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf84ab700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf007d6e0 con 0x7f4bf01099e0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf6247700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf6247700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32920/0 (socket says 172.21.15.156:32920) 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf6247700 1 -- 172.21.15.156:0/837064064 learned_addr learned my addr 172.21.15.156:0/837064064 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf6247700 1 -- 172.21.15.156:0/837064064 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf010c310 con 0x7f4bf01099e0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf6247700 1 --2- 172.21.15.156:0/837064064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=799e15ac0ccb0cb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf5245700 1 -- 172.21.15.156:0/837064064 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0004160 con 0x7f4bf01099e0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf5245700 1 -- 172.21.15.156:0/837064064 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4be0004300 con 0x7f4bf01099e0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.086+0000 7f4bf5245700 1 -- 172.21.15.156:0/837064064 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0011af0 con 0x7f4bf01099e0 2024-05-26T11:52:46.215 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 -- 172.21.15.156:0/837064064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 msgr2=0x7f4bf010bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/837064064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f4be00098a0 tx=0x7f4be0009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 -- 172.21.15.156:0/837064064 shutdown_connections 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/837064064 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf010bdd0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 -- 172.21.15.156:0/837064064 >> 172.21.15.156:0/837064064 conn(0x7f4bf0103450 msgr2=0x7f4bf0105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 -- 172.21.15.156:0/837064064 shutdown_connections 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.087+0000 7f4bf84ab700 1 -- 172.21.15.156:0/837064064 wait complete. 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf84ab700 1 Processor -- start 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf84ab700 1 -- start start 2024-05-26T11:52:46.216 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf84ab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf84ab700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4bf010d280 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf6247700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf6247700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32932/0 (socket says 172.21.15.156:32932) 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf6247700 1 -- 172.21.15.156:0/3125752407 learned_addr learned my addr 172.21.15.156:0/3125752407 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf6247700 1 -- 172.21.15.156:0/3125752407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bf019d3f0 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.088+0000 7f4bf6247700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009fb0 tx=0x7f4be0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.089+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be00036a0 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.089+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4be001a430 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.089+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bf019d680 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.089+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4be0019550 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.089+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bf01a0370 con 0x7f4bf01099e0 2024-05-26T11:52:46.217 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f4be0003840 con 0x7f4bf01099e0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bf0064ba0 con 0x7f4bf01099e0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4be77fe700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 0x7f4bdc040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4be0053810 con 0x7f4bf01099e0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4bf5a46700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 0x7f4bdc040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.090+0000 7f4bf5a46700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 0x7f4bdc040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.095+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4be000b3e0 con 0x7f4bf01099e0 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.303+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f4bf01046b0 con 0x7f4bdc03e470 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.622+0000 7f4be77fe700 1 -- 172.21.15.156:0/3125752407 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f4bf01046b0 con 0x7f4bdc03e470 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 msgr2=0x7f4bdc040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 0x7f4bdc040930 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4bec006fd0 tx=0x7f4bec006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 msgr2=0x7f4bf019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:46.218 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4be0009fb0 tx=0x7f4be0011940 comp rx=0 tx=0).stop 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 shutdown_connections 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f4bdc03e470 0x7f4bdc040930 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 --2- 172.21.15.156:0/3125752407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4bf01099e0 0x7f4bf019ceb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 >> 172.21.15.156:0/3125752407 conn(0x7f4bf0103450 msgr2=0x7f4bf0103fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 shutdown_connections 2024-05-26T11:52:46.219 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:45.624+0000 7f4bf84ab700 1 -- 172.21.15.156:0/3125752407 wait complete. 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc080913700 1 Processor -- start 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc080913700 1 -- start start 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc080913700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc080913700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c108790 con 0x7fc07c107e30 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc07a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc07a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32936/0 (socket says 172.21.15.156:32936) 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.618+0000 7fc07a59c700 1 -- 172.21.15.156:0/2515003812 learned_addr learned my addr 172.21.15.156:0/2515003812 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:47.422 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.619+0000 7fc07a59c700 1 -- 172.21.15.156:0/2515003812 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c108f90 con 0x7fc07c107e30 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.619+0000 7fc07a59c700 1 --2- 172.21.15.156:0/2515003812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00bb40 tx=0x7fc06c00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a80f30ce5a603f1f server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.619+0000 7fc07959a700 1 -- 172.21.15.156:0/2515003812 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c00dbe0 con 0x7fc07c107e30 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.619+0000 7fc07959a700 1 -- 172.21.15.156:0/2515003812 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc06c0044c0 con 0x7fc07c107e30 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.619+0000 7fc07959a700 1 -- 172.21.15.156:0/2515003812 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c010470 con 0x7fc07c107e30 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- 172.21.15.156:0/2515003812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 msgr2=0x7fc07c108250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 --2- 172.21.15.156:0/2515003812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00bb40 tx=0x7fc06c00be50 comp rx=0 tx=0).stop 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- 172.21.15.156:0/2515003812 shutdown_connections 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 --2- 172.21.15.156:0/2515003812 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c108250 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- 172.21.15.156:0/2515003812 >> 172.21.15.156:0/2515003812 conn(0x7fc07c1033d0 msgr2=0x7fc07c105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- 172.21.15.156:0/2515003812 shutdown_connections 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- 172.21.15.156:0/2515003812 wait complete. 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 Processor -- start 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.620+0000 7fc080913700 1 -- start start 2024-05-26T11:52:47.423 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc080913700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc080913700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc07c07c860 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc07a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc07a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32942/0 (socket says 172.21.15.156:32942) 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc07a59c700 1 -- 172.21.15.156:0/904478906 learned_addr learned my addr 172.21.15.156:0/904478906 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc07a59c700 1 -- 172.21.15.156:0/904478906 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc07c19b170 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc07a59c700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00d650 tx=0x7fc06c003ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c027070 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc06c010c00 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc07c19b400 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc06c022ba0 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.621+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc07c19e0f0 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.622+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc06c02a070 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.622+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc07c064ba0 con 0x7fc07c107e30 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.623+0000 7fc06b7fe700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 0x7fc064040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:47.424 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.623+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc06c01f070 con 0x7fc07c107e30 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.623+0000 7fc079d9b700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 0x7fc064040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.623+0000 7fc079d9b700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 0x7fc064040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc070006fd0 tx=0x7fc070006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.628+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc06c015070 con 0x7fc07c107e30 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.833+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fc07c0008d0 con 0x7fc06403e470 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.834+0000 7fc06b7fe700 1 -- 172.21.15.156:0/904478906 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fc07c0008d0 con 0x7fc06403e470 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 msgr2=0x7fc064040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 0x7fc064040930 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc070006fd0 tx=0x7fc070006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 msgr2=0x7fc07c19ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc06c00d650 tx=0x7fc06c003ff0 comp rx=0 tx=0).stop 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 shutdown_connections 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc06403e470 0x7fc064040930 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 --2- 172.21.15.156:0/904478906 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc07c107e30 0x7fc07c19ac30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 >> 172.21.15.156:0/904478906 conn(0x7fc07c1033d0 msgr2=0x7fc07c1917f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 shutdown_connections 2024-05-26T11:52:47.425 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:46.836+0000 7fc080913700 1 -- 172.21.15.156:0/904478906 wait complete. 2024-05-26T11:52:47.426 INFO:teuthology.orchestra.run.smithi156.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-26T11:52:47.426 INFO:teuthology.orchestra.run.smithi156.stdout:Adding key to root@localhost authorized_keys... 2024-05-26T11:52:47.426 INFO:teuthology.orchestra.run.smithi156.stdout:Adding host smithi156... 2024-05-26T11:52:47.733 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:47 smithi156 ceph-mon[32025]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:48.639 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:48 smithi156 ceph-mon[32025]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi156", "addr": "172.21.15.156", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:49.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:49 smithi156 ceph-mon[32025]: Deploying cephadm binary to smithi156 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout Added host 'smithi156' with addr '172.21.15.156' 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc02341c700 1 Processor -- start 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc02341c700 1 -- start start 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc02341c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc02341c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c07d700 con 0x7fc01c1099e0 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc0211b8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc0211b8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32946/0 (socket says 172.21.15.156:32946) 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.836+0000 7fc0211b8700 1 -- 172.21.15.156:0/2977447274 learned_addr learned my addr 172.21.15.156:0/2977447274 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:52.904 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.837+0000 7fc0211b8700 1 -- 172.21.15.156:0/2977447274 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c10c310 con 0x7fc01c1099e0 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.837+0000 7fc0211b8700 1 --2- 172.21.15.156:0/2977447274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0180098a0 tx=0x7fc018009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6676e6647645b7e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.837+0000 7fc013fff700 1 -- 172.21.15.156:0/2977447274 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc018004160 con 0x7fc01c1099e0 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.837+0000 7fc013fff700 1 -- 172.21.15.156:0/2977447274 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc018004300 con 0x7fc01c1099e0 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.837+0000 7fc013fff700 1 -- 172.21.15.156:0/2977447274 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc018011af0 con 0x7fc01c1099e0 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- 172.21.15.156:0/2977447274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 msgr2=0x7fc01c10bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 --2- 172.21.15.156:0/2977447274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc0180098a0 tx=0x7fc018009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- 172.21.15.156:0/2977447274 shutdown_connections 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 --2- 172.21.15.156:0/2977447274 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c10bdd0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- 172.21.15.156:0/2977447274 >> 172.21.15.156:0/2977447274 conn(0x7fc01c103470 msgr2=0x7fc01c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- 172.21.15.156:0/2977447274 shutdown_connections 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- 172.21.15.156:0/2977447274 wait complete. 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 Processor -- start 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- start start 2024-05-26T11:52:52.905 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.838+0000 7fc02341c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc01c10d2d0 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0211b8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0211b8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32952/0 (socket says 172.21.15.156:32952) 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0211b8700 1 -- 172.21.15.156:0/3794758431 learned_addr learned my addr 172.21.15.156:0/3794758431 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0211b8700 1 -- 172.21.15.156:0/3794758431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01c19d3c0 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0211b8700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc018011ac0 tx=0x7fc018011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc018003970 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc018003b10 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc01c19d650 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc018024610 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.839+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01c1a0340 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.840+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7fc018024840 con 0x7fc01c1099e0 2024-05-26T11:52:52.906 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.840+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc01c064ba0 con 0x7fc01c1099e0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.840+0000 7fc0127fc700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 0x7fc008040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.840+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc018053880 con 0x7fc01c1099e0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.840+0000 7fc0209b7700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 0x7fc008040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.841+0000 7fc0209b7700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 0x7fc008040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006fd0 tx=0x7fc00c006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:47.845+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc01802c350 con 0x7fc01c1099e0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:48.055+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi156", "addr": "172.21.15.156", "target": ["mon-mgr", ""]}) -- 0x7fc01c104650 con 0x7fc00803e4c0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.131+0000 7fc0127fc700 1 -- 172.21.15.156:0/3794758431 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc01c104650 con 0x7fc00803e4c0 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 msgr2=0x7fc008040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 0x7fc008040980 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc00c006fd0 tx=0x7fc00c006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 msgr2=0x7fc01c19ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc018011ac0 tx=0x7fc018011ba0 comp rx=0 tx=0).stop 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 shutdown_connections 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fc00803e4c0 0x7fc008040980 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 --2- 172.21.15.156:0/3794758431 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc01c1099e0 0x7fc01c19ce80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:52.907 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 >> 172.21.15.156:0/3794758431 conn(0x7fc01c103470 msgr2=0x7fc01c103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:52.908 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 shutdown_connections 2024-05-26T11:52:52.908 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:52.134+0000 7fc02341c700 1 -- 172.21.15.156:0/3794758431 wait complete. 2024-05-26T11:52:52.908 INFO:teuthology.orchestra.run.smithi156.stdout:Deploying unmanaged mon service... 2024-05-26T11:52:53.200 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:53 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:53.200 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:53 smithi156 ceph-mon[32025]: Added host smithi156 2024-05-26T11:52:53.200 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:53 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.302+0000 7f428b8d8700 1 Processor -- start 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.302+0000 7f428b8d8700 1 -- start start 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f428b8d8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f428b8d8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f428407d6e0 con 0x7f42841077d0 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f4289674700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f4289674700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32962/0 (socket says 172.21.15.156:32962) 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f4289674700 1 -- 172.21.15.156:0/3455644170 learned_addr learned my addr 172.21.15.156:0/3455644170 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:54.355 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f4289674700 1 -- 172.21.15.156:0/3455644170 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4284108130 con 0x7f42841077d0 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f4289674700 1 --2- 172.21.15.156:0/3455644170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4280009e80 tx=0x7f428000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72023bee99521ba1 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f427bfff700 1 -- 172.21.15.156:0/3455644170 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4280017070 con 0x7f42841077d0 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f427bfff700 1 -- 172.21.15.156:0/3455644170 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f428000b8c0 con 0x7f42841077d0 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.303+0000 7f427bfff700 1 -- 172.21.15.156:0/3455644170 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4280004490 con 0x7f42841077d0 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 -- 172.21.15.156:0/3455644170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 msgr2=0x7f4284107bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 --2- 172.21.15.156:0/3455644170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4280009e80 tx=0x7f428000b330 comp rx=0 tx=0).stop 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 -- 172.21.15.156:0/3455644170 shutdown_connections 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 --2- 172.21.15.156:0/3455644170 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f4284107bf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 -- 172.21.15.156:0/3455644170 >> 172.21.15.156:0/3455644170 conn(0x7f4284103450 msgr2=0x7f4284105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 -- 172.21.15.156:0/3455644170 shutdown_connections 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.304+0000 7f428b8d8700 1 -- 172.21.15.156:0/3455644170 wait complete. 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f428b8d8700 1 Processor -- start 2024-05-26T11:52:54.356 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f428b8d8700 1 -- start start 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f428b8d8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f428b8d8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4284109160 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f4289674700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f4289674700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:32974/0 (socket says 172.21.15.156:32974) 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f4289674700 1 -- 172.21.15.156:0/4163669926 learned_addr learned my addr 172.21.15.156:0/4163669926 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f4289674700 1 -- 172.21.15.156:0/4163669926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f428419f580 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f4289674700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f428000b7b0 tx=0x7f428000be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4280020020 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428419f810 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f4280012430 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.305+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4280017040 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.306+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42841a2500 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.306+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f4280023020 con 0x7f42841077d0 2024-05-26T11:52:54.357 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.307+0000 7f427a7fc700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 0x7f42700408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.307+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4284064ba0 con 0x7f42841077d0 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.307+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f428000f050 con 0x7f42841077d0 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.307+0000 7f4288e73700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 0x7f42700408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.307+0000 7f4288e73700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 0x7f42700408e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4274006fd0 tx=0x7f4274006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.312+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f428002a020 con 0x7f42841077d0 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.558+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f428402e910 con 0x7f427003e420 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.563+0000 7f427a7fc700 1 -- 172.21.15.156:0/4163669926 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f428402e910 con 0x7f427003e420 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 msgr2=0x7f42700408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 0x7f42700408e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4274006fd0 tx=0x7f4274006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:54.358 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 msgr2=0x7f428419f040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:54.361 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f428000b7b0 tx=0x7f428000be20 comp rx=0 tx=0).stop 2024-05-26T11:52:54.361 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 shutdown_connections 2024-05-26T11:52:54.361 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f427003e420 0x7f42700408e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:54.361 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 --2- 172.21.15.156:0/4163669926 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f42841077d0 0x7f428419f040 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:54.361 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 >> 172.21.15.156:0/4163669926 conn(0x7f4284103450 msgr2=0x7f4284195c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:54.362 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 shutdown_connections 2024-05-26T11:52:54.362 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:53.565+0000 7f428b8d8700 1 -- 172.21.15.156:0/4163669926 wait complete. 2024-05-26T11:52:54.362 INFO:teuthology.orchestra.run.smithi156.stdout:Deploying unmanaged mgr service... 2024-05-26T11:52:54.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:54 smithi156 ceph-mon[32025]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:54.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:54 smithi156 ceph-mon[32025]: Saving service mon spec with placement count:5 2024-05-26T11:52:54.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:54 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:55.798 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.766+0000 7f71f3e26700 1 Processor -- start 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.766+0000 7f71f3e26700 1 -- start start 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.766+0000 7f71f3e26700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.766+0000 7f71f3e26700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71ec07d740 con 0x7f71ec1099a0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f1bc2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f1bc2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60726/0 (socket says 172.21.15.156:60726) 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f1bc2700 1 -- 172.21.15.156:0/187805379 learned_addr learned my addr 172.21.15.156:0/187805379 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f1bc2700 1 -- 172.21.15.156:0/187805379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ec10c2d0 con 0x7f71ec1099a0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f1bc2700 1 --2- 172.21.15.156:0/187805379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71e80098a0 tx=0x7f71e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc5411317554c78 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f0bc0700 1 -- 172.21.15.156:0/187805379 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71e8004160 con 0x7f71ec1099a0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f0bc0700 1 -- 172.21.15.156:0/187805379 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f71e8004300 con 0x7f71ec1099a0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.767+0000 7f71f0bc0700 1 -- 172.21.15.156:0/187805379 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71e8011af0 con 0x7f71ec1099a0 2024-05-26T11:52:55.799 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 -- 172.21.15.156:0/187805379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 msgr2=0x7f71ec10bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 --2- 172.21.15.156:0/187805379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f71e80098a0 tx=0x7f71e8009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 -- 172.21.15.156:0/187805379 shutdown_connections 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 --2- 172.21.15.156:0/187805379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec10bd90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 -- 172.21.15.156:0/187805379 >> 172.21.15.156:0/187805379 conn(0x7f71ec103470 msgr2=0x7f71ec105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 -- 172.21.15.156:0/187805379 shutdown_connections 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.768+0000 7f71f3e26700 1 -- 172.21.15.156:0/187805379 wait complete. 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f3e26700 1 Processor -- start 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f3e26700 1 -- start start 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f3e26700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f3e26700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71ec10d2a0 con 0x7f71ec1099a0 2024-05-26T11:52:55.800 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f1bc2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f1bc2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60742/0 (socket says 172.21.15.156:60742) 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f1bc2700 1 -- 172.21.15.156:0/2905979394 learned_addr learned my addr 172.21.15.156:0/2905979394 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f1bc2700 1 -- 172.21.15.156:0/2905979394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ec109760 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.769+0000 7f71f1bc2700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71e8011ac0 tx=0x7f71e8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.770+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71e8003970 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.770+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f71e8003b10 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.770+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ec107950 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.770+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f71e8024610 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.770+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ec107e90 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f71e8024890 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71ec064ba0 con 0x7f71ec1099a0 2024-05-26T11:52:55.801 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71e2ffd700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 0x7f71d8040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f71e8053ab0 con 0x7f71ec1099a0 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71f13c1700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 0x7f71d8040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.771+0000 7f71f13c1700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 0x7f71d8040930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f71dc006fd0 tx=0x7f71dc006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.776+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f71e802c350 con 0x7f71ec1099a0 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:54.999+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f71ec1046e0 con 0x7f71d803e470 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.004+0000 7f71e2ffd700 1 -- 172.21.15.156:0/2905979394 <== mgr.14118 v2:172.21.15.156:6800/2588638081 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f71ec1046e0 con 0x7f71d803e470 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 msgr2=0x7f71d8040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 0x7f71d8040930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f71dc006fd0 tx=0x7f71dc006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 msgr2=0x7f71ec109220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f71e8011ac0 tx=0x7f71e8011ba0 comp rx=0 tx=0).stop 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 shutdown_connections 2024-05-26T11:52:55.802 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f71d803e470 0x7f71d8040930 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:55.803 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 --2- 172.21.15.156:0/2905979394 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ec1099a0 0x7f71ec109220 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:55.803 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.006+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 >> 172.21.15.156:0/2905979394 conn(0x7f71ec103470 msgr2=0x7f71ec103fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:55.803 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.007+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 shutdown_connections 2024-05-26T11:52:55.803 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:55.007+0000 7f71f3e26700 1 -- 172.21.15.156:0/2905979394 wait complete. 2024-05-26T11:52:56.193 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:56 smithi156 ceph-mon[32025]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:52:56.193 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:56 smithi156 ceph-mon[32025]: Saving service mgr spec with placement count:2 2024-05-26T11:52:56.193 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:56 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:56.912 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f791b2e5700 1 Processor -- start 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f791b2e5700 1 -- start start 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f791b2e5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f791b2e5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7914109890 con 0x7f7914108f30 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f7919081700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f7919081700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60750/0 (socket says 172.21.15.156:60750) 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.211+0000 7f7919081700 1 -- 172.21.15.156:0/782459186 learned_addr learned my addr 172.21.15.156:0/782459186 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.212+0000 7f7919081700 1 -- 172.21.15.156:0/782459186 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791410a090 con 0x7f7914108f30 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.212+0000 7f7919081700 1 --2- 172.21.15.156:0/782459186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79040098a0 tx=0x7f7904009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a89be8d005367644 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.212+0000 7f790bfff700 1 -- 172.21.15.156:0/782459186 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904004160 con 0x7f7914108f30 2024-05-26T11:52:56.913 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.212+0000 7f790bfff700 1 -- 172.21.15.156:0/782459186 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f7904004300 con 0x7f7914108f30 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.212+0000 7f790bfff700 1 -- 172.21.15.156:0/782459186 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904011af0 con 0x7f7914108f30 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- 172.21.15.156:0/782459186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 msgr2=0x7f7914109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 --2- 172.21.15.156:0/782459186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f79040098a0 tx=0x7f7904009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- 172.21.15.156:0/782459186 shutdown_connections 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 --2- 172.21.15.156:0/782459186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f7914109350 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- 172.21.15.156:0/782459186 >> 172.21.15.156:0/782459186 conn(0x7f7914104550 msgr2=0x7f7914106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- 172.21.15.156:0/782459186 shutdown_connections 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- 172.21.15.156:0/782459186 wait complete. 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 Processor -- start 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- start start 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.213+0000 7f791b2e5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f791410d4d0 con 0x7f7914108f30 2024-05-26T11:52:56.914 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f7919081700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f7919081700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60758/0 (socket says 172.21.15.156:60758) 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f7919081700 1 -- 172.21.15.156:0/1876140793 learned_addr learned my addr 172.21.15.156:0/1876140793 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f7919081700 1 -- 172.21.15.156:0/1876140793 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791419d400 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f7919081700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7904009fb0 tx=0x7f7904011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f79040036a0 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f790401a430 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f791419d690 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7904019550 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.214+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79141a0380 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.215+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f7904003840 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.215+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7914064ba0 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.215+0000 7f790a7fc700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 0x7f79000408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.215+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f79040538b0 con 0x7f7914108f30 2024-05-26T11:52:56.915 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.215+0000 7f7918880700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 0x7f79000408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.216+0000 7f7918880700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 0x7f79000408e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7910006fd0 tx=0x7f7910006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.220+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7904021e80 con 0x7f7914108f30 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.423+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f79141a09d0 con 0x7f7914108f30 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.425+0000 7f790a7fc700 1 -- 172.21.15.156:0/1876140793 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v6)=0 v6) ==== 142+0+0 (secure 0 0 0) 0x7f7904003b30 con 0x7f7914108f30 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 msgr2=0x7f79000408e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 0x7f79000408e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f7910006fd0 tx=0x7f7910006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 msgr2=0x7f791419cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f7904009fb0 tx=0x7f7904011940 comp rx=0 tx=0).stop 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 shutdown_connections 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f790003e420 0x7f79000408e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 --2- 172.21.15.156:0/1876140793 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7914108f30 0x7f791419cec0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 >> 172.21.15.156:0/1876140793 conn(0x7f7914104550 msgr2=0x7f7914105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:56.916 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 shutdown_connections 2024-05-26T11:52:56.917 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:56.428+0000 7f791b2e5700 1 -- 172.21.15.156:0/1876140793 wait complete. 2024-05-26T11:52:57.519 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:57 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1876140793' entity='client.admin' 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.288+0000 7f6724d14700 1 Processor -- start 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.288+0000 7f6724d14700 1 -- start start 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.288+0000 7f6724d14700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.288+0000 7f6724d14700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f67201098d0 con 0x7f6720108f70 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:58.230 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60768/0 (socket says 172.21.15.156:60768) 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671e59c700 1 -- 172.21.15.156:0/1667858600 learned_addr learned my addr 172.21.15.156:0/1667858600 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671e59c700 1 -- 172.21.15.156:0/1667858600 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672010a080 con 0x7f6720108f70 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671e59c700 1 --2- 172.21.15.156:0/1667858600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f02fb28bd76d62 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671d59a700 1 -- 172.21.15.156:0/1667858600 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710004160 con 0x7f6720108f70 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671d59a700 1 -- 172.21.15.156:0/1667858600 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6710004300 con 0x7f6720108f70 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.289+0000 7f671d59a700 1 -- 172.21.15.156:0/1667858600 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710011af0 con 0x7f6720108f70 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 -- 172.21.15.156:0/1667858600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 msgr2=0x7f6720109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 --2- 172.21.15.156:0/1667858600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f67100098a0 tx=0x7f6710009bb0 comp rx=0 tx=0).stop 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 -- 172.21.15.156:0/1667858600 shutdown_connections 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 --2- 172.21.15.156:0/1667858600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f6720109390 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 -- 172.21.15.156:0/1667858600 >> 172.21.15.156:0/1667858600 conn(0x7f6720104530 msgr2=0x7f6720106950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 -- 172.21.15.156:0/1667858600 shutdown_connections 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.290+0000 7f6724d14700 1 -- 172.21.15.156:0/1667858600 wait complete. 2024-05-26T11:52:58.231 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f6724d14700 1 Processor -- start 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f6724d14700 1 -- start start 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f6724d14700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f6724d14700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f672010d4c0 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f671e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f671e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60770/0 (socket says 172.21.15.156:60770) 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f671e59c700 1 -- 172.21.15.156:0/681925704 learned_addr learned my addr 172.21.15.156:0/681925704 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f671e59c700 1 -- 172.21.15.156:0/681925704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672019d3f0 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.291+0000 7f671e59c700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6710011ac0 tx=0x7f6710011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.292+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710003970 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.292+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f672019d680 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.292+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6710003b10 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.292+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6710024610 con 0x7f6720108f70 2024-05-26T11:52:58.232 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.292+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67201a0370 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f6710024890 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6720064ba0 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f670f7fe700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 0x7f6708040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f67100538c0 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f671dd9b700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 0x7f6708040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.293+0000 7f671dd9b700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 0x7f6708040930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6714006fd0 tx=0x7f6714006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.298+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f671002c350 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.502+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f67201a0540 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.504+0000 7f670f7fe700 1 -- 172.21.15.156:0/681925704 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v7)=0 v7) ==== 130+0+0 (secure 0 0 0) 0x7f67201a0540 con 0x7f6720108f70 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 msgr2=0x7f6708040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 0x7f6708040930 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f6714006fd0 tx=0x7f6714006e40 comp rx=0 tx=0).stop 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 msgr2=0x7f672019ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6710011ac0 tx=0x7f6710011ba0 comp rx=0 tx=0).stop 2024-05-26T11:52:58.233 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 shutdown_connections 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f670803e470 0x7f6708040930 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 --2- 172.21.15.156:0/681925704 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6720108f70 0x7f672019ceb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 >> 172.21.15.156:0/681925704 conn(0x7f6720104530 msgr2=0x7f6720105090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 shutdown_connections 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:57.507+0000 7f6724d14700 1 -- 172.21.15.156:0/681925704 wait complete. 2024-05-26T11:52:58.234 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling the dashboard module... 2024-05-26T11:52:58.578 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:58 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/681925704' entity='client.admin' 2024-05-26T11:52:58.578 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:58 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:58.579 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:58 smithi156 ceph-mon[32025]: from='mgr.14118 172.21.15.156:0/1489268206' entity='mgr.x' 2024-05-26T11:52:59.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:52:59 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/565729902' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-26T11:52:59.795 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: ignoring --setuser ceph since I am not root 2024-05-26T11:52:59.795 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: ignoring --setgroup ceph since I am not root 2024-05-26T11:52:59.795 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:59.692+0000 7f71db017200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T11:53:00.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:59.879+0000 7f71db017200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:53:00.109 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:52:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:52:59.984+0000 7f71db017200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T11:53:00.111 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f903979e700 1 Processor -- start 2024-05-26T11:53:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f903979e700 1 -- start start 2024-05-26T11:53:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f903979e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f903979e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9034107a30 con 0x7f9034107040 2024-05-26T11:53:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f9033fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:00.116 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f9033fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60776/0 (socket says 172.21.15.156:60776) 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.712+0000 7f9033fff700 1 -- 172.21.15.156:0/170430938 learned_addr learned my addr 172.21.15.156:0/170430938 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.713+0000 7f9033fff700 1 -- 172.21.15.156:0/170430938 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9034109fd0 con 0x7f9034107040 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.713+0000 7f9033fff700 1 --2- 172.21.15.156:0/170430938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4963c0f1c292ac2f server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.713+0000 7f9032ffd700 1 -- 172.21.15.156:0/170430938 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9024004160 con 0x7f9034107040 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.713+0000 7f9032ffd700 1 -- 172.21.15.156:0/170430938 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9024004300 con 0x7f9034107040 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.713+0000 7f9032ffd700 1 -- 172.21.15.156:0/170430938 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9024011a20 con 0x7f9034107040 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 -- 172.21.15.156:0/170430938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 msgr2=0x7f9034107460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 --2- 172.21.15.156:0/170430938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f90240098a0 tx=0x7f9024009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 -- 172.21.15.156:0/170430938 shutdown_connections 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 --2- 172.21.15.156:0/170430938 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f9034107460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 -- 172.21.15.156:0/170430938 >> 172.21.15.156:0/170430938 conn(0x7f9034102970 msgr2=0x7f9034104db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 -- 172.21.15.156:0/170430938 shutdown_connections 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.714+0000 7f903979e700 1 -- 172.21.15.156:0/170430938 wait complete. 2024-05-26T11:53:00.117 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.715+0000 7f903979e700 1 Processor -- start 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.715+0000 7f903979e700 1 -- start start 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.715+0000 7f903979e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.715+0000 7f903979e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f903410d380 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f9033fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f9033fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60784/0 (socket says 172.21.15.156:60784) 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f9033fff700 1 -- 172.21.15.156:0/565729902 learned_addr learned my addr 172.21.15.156:0/565729902 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f9033fff700 1 -- 172.21.15.156:0/565729902 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f903419d200 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f9033fff700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90240119f0 tx=0x7f9024011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90240039e0 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f902401b8f0 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f903419d490 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.716+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f90240237a0 con 0x7f9034107040 2024-05-26T11:53:00.118 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.717+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90341a0180 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.717+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 8) ==== 50222+0+0 (secure 0 0 0) 0x7f90240193d0 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.718+0000 7f90317fa700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 0x7f901c040930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.718+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f90240556f0 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.718+0000 7f90337fe700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 0x7f901c040930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.718+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f903406a680 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.719+0000 7f90337fe700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 0x7f901c040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9028006fd0 tx=0x7f9028006e40 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:58.727+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f902401f030 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.005+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f90341a0700 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.507+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f9024021300 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.509+0000 7f90317fa700 1 -- 172.21.15.156:0/565729902 <== mon.0 v2:172.21.15.156:3300/0 8 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f9024016d20 con 0x7f9034107040 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.510+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 msgr2=0x7f901c040930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:00.119 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 0x7f901c040930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f9028006fd0 tx=0x7f9028006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 msgr2=0x7f903419ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f90240119f0 tx=0x7f9024011ad0 comp rx=0 tx=0).stop 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 shutdown_connections 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f901c03e470 0x7f901c040930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 --2- 172.21.15.156:0/565729902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9034107040 0x7f903419ccc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 >> 172.21.15.156:0/565729902 conn(0x7f9034102970 msgr2=0x7f9034103450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 shutdown_connections 2024-05-26T11:53:00.120 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:52:59.511+0000 7f903979e700 1 -- 172.21.15.156:0/565729902 wait complete. 2024-05-26T11:53:00.362 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:00 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:00.303+0000 7f71db017200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T11:53:00.779 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:00 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/565729902' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-26T11:53:00.780 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:00 smithi156 ceph-mon[32025]: mgrmap e9: x(active, since 18s) 2024-05-26T11:53:00.780 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:00 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:00.393+0000 7f71db017200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-26T11:53:01.430 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.573+0000 7f373ba1a700 1 Processor -- start 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.573+0000 7f373ba1a700 1 -- start start 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.573+0000 7f373ba1a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.573+0000 7f373ba1a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3734109890 con 0x7f3734108f30 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f37397b6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f37397b6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60804/0 (socket says 172.21.15.156:60804) 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f37397b6700 1 -- 172.21.15.156:0/4116742600 learned_addr learned my addr 172.21.15.156:0/4116742600 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f37397b6700 1 -- 172.21.15.156:0/4116742600 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373410a090 con 0x7f3734108f30 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f37397b6700 1 --2- 172.21.15.156:0/4116742600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37240098a0 tx=0x7f3724009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327f9c70e5f1c2c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f372bfff700 1 -- 172.21.15.156:0/4116742600 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3724004160 con 0x7f3734108f30 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f372bfff700 1 -- 172.21.15.156:0/4116742600 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3724004300 con 0x7f3734108f30 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.574+0000 7f372bfff700 1 -- 172.21.15.156:0/4116742600 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3724011a20 con 0x7f3734108f30 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 -- 172.21.15.156:0/4116742600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 msgr2=0x7f3734109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:01.431 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 --2- 172.21.15.156:0/4116742600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f37240098a0 tx=0x7f3724009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 -- 172.21.15.156:0/4116742600 shutdown_connections 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 --2- 172.21.15.156:0/4116742600 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f3734109350 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 -- 172.21.15.156:0/4116742600 >> 172.21.15.156:0/4116742600 conn(0x7f3734104550 msgr2=0x7f3734106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 -- 172.21.15.156:0/4116742600 shutdown_connections 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.575+0000 7f373ba1a700 1 -- 172.21.15.156:0/4116742600 wait complete. 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f373ba1a700 1 Processor -- start 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f373ba1a700 1 -- start start 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f373ba1a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f373ba1a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f373410d4d0 con 0x7f3734108f30 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f37397b6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f37397b6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60812/0 (socket says 172.21.15.156:60812) 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.576+0000 7f37397b6700 1 -- 172.21.15.156:0/3006912680 learned_addr learned my addr 172.21.15.156:0/3006912680 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f37397b6700 1 -- 172.21.15.156:0/3006912680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373407f940 con 0x7f3734108f30 2024-05-26T11:53:01.432 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f37397b6700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37240116e0 tx=0x7f3724011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3724003b20 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f372401b8f0 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373407db30 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37240238c0 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.577+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373407e070 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7f3724023a60 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3734064ba0 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f372a7fc700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 0x7f3720040990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3724053fe0 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f3738fb5700 1 -- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 msgr2=0x7f3720040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.578+0000 7f3738fb5700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 0x7f3720040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.583+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f372401f030 con 0x7f3734108f30 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.779+0000 7f3738fb5700 1 -- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 msgr2=0x7f3720040990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:01.433 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.779+0000 7f3738fb5700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 0x7f3720040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.880+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f373407e9a0 con 0x7f3734108f30 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.881+0000 7f372a7fc700 1 -- 172.21.15.156:0/3006912680 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f372400b3e0 con 0x7f3734108f30 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 msgr2=0x7f3720040990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 0x7f3720040990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 msgr2=0x7f373407f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f37240116e0 tx=0x7f3724011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 shutdown_connections 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7f372003e4d0 0x7f3720040990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 --2- 172.21.15.156:0/3006912680 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3734108f30 0x7f373407f400 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 >> 172.21.15.156:0/3006912680 conn(0x7f3734104550 msgr2=0x7f3734104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 shutdown_connections 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:00.883+0000 7f373ba1a700 1 -- 172.21.15.156:0/3006912680 wait complete. 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for the mgr to restart... 2024-05-26T11:53:01.434 INFO:teuthology.orchestra.run.smithi156.stdout:Waiting for mgr epoch 9... 2024-05-26T11:53:01.762 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:01 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3006912680' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-26T11:53:01.762 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:01.635+0000 7f71db017200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T11:53:02.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:01.816+0000 7f71db017200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T11:53:02.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:01.991+0000 7f71db017200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T11:53:02.379 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:02.285+0000 7f71db017200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T11:53:02.783 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:02.378+0000 7f71db017200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T11:53:02.783 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:02.585+0000 7f71db017200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T11:53:03.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:02.783+0000 7f71db017200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:53:03.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:03.393+0000 7f71db017200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T11:53:04.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:03.616+0000 7f71db017200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-26T11:53:04.519 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:04.046+0000 7f71db017200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T11:53:04.519 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:04.144+0000 7f71db017200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T11:53:04.519 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:04.518+0000 7f71db017200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T11:53:05.221 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:04.879+0000 7f71db017200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T11:53:05.221 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:04.973+0000 7f71db017200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T11:53:05.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:05.220+0000 7f71db017200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T11:53:05.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:05.356+0000 7f71db017200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T11:53:05.544 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:05.455+0000 7f71db017200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T11:53:06.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:05.785+0000 7f71db017200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T11:53:06.044 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:05.879+0000 7f71db017200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T11:53:06.760 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:06.485+0000 7f71db017200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T11:53:06.760 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 11:53:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T11:53:06.705+0000 7f71db017200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T11:53:07.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: Active manager daemon x restarted 2024-05-26T11:53:07.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: Activating manager daemon x 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: osdmap e3: 0 total, 0 up, 0 in 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: mgrmap e10: x(active, starting, since 0.00352955s) 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: Manager daemon x is now available 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:07.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: [26/May/2024:11:53:07] ENGINE Bus STARTING 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: [26/May/2024:11:53:07] ENGINE Serving on https://172.21.15.156:7150 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: [26/May/2024:11:53:07] ENGINE Serving on http://172.21.15.156:8765 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: [26/May/2024:11:53:07] ENGINE Bus STARTED 2024-05-26T11:53:07.919 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:07 smithi156 ceph-mon[32025]: mgrmap e11: x(active, since 1.00638s) 2024-05-26T11:53:08.482 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout { 2024-05-26T11:53:08.482 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-26T11:53:08.482 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-26T11:53:08.482 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout } 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.861+0000 7fe82fe2f700 1 Processor -- start 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.861+0000 7fe82fe2f700 1 -- start start 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82fe2f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82fe2f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe828109e20 con 0x7fe828109430 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82dbcb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82dbcb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60818/0 (socket says 172.21.15.156:60818) 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82dbcb700 1 -- 172.21.15.156:0/2323355055 learned_addr learned my addr 172.21.15.156:0/2323355055 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82dbcb700 1 -- 172.21.15.156:0/2323355055 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe828109fa0 con 0x7fe828109430 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82dbcb700 1 --2- 172.21.15.156:0/2323355055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098a0 tx=0x7fe818009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7564c435ff8a1a45 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82cbc9700 1 -- 172.21.15.156:0/2323355055 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe818004160 con 0x7fe828109430 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82cbc9700 1 -- 172.21.15.156:0/2323355055 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe818004300 con 0x7fe828109430 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.862+0000 7fe82cbc9700 1 -- 172.21.15.156:0/2323355055 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe818011a20 con 0x7fe828109430 2024-05-26T11:53:08.483 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 -- 172.21.15.156:0/2323355055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 msgr2=0x7fe828109850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/2323355055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe8180098a0 tx=0x7fe818009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 -- 172.21.15.156:0/2323355055 shutdown_connections 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/2323355055 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe828109850 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 -- 172.21.15.156:0/2323355055 >> 172.21.15.156:0/2323355055 conn(0x7fe828102ba0 msgr2=0x7fe828104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 -- 172.21.15.156:0/2323355055 shutdown_connections 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.863+0000 7fe82fe2f700 1 -- 172.21.15.156:0/2323355055 wait complete. 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82fe2f700 1 Processor -- start 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82fe2f700 1 -- start start 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82fe2f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82fe2f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe828107df0 con 0x7fe828109430 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82dbcb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82dbcb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60822/0 (socket says 172.21.15.156:60822) 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82dbcb700 1 -- 172.21.15.156:0/3377249419 learned_addr learned my addr 172.21.15.156:0/3377249419 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.864+0000 7fe82dbcb700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe82819f530 con 0x7fe828109430 2024-05-26T11:53:08.484 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe82dbcb700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe818004660 tx=0x7fe818011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe818003aa0 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe82819f820 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe81801b8f0 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe818023900 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.865+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8281a2460 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 9) ==== 50235+0+0 (secure 0 0 0) 0x7fe818023bc0 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe81effd700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe814041090 con 0x7fe81403e4c0 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe818053ef0 con 0x7fe828109430 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe82d3ca700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:01.866+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:02.067+0000 7fe82d3ca700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:02.067+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:02.468+0000 7fe82d3ca700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:08.485 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:02.468+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:03.269+0000 7fe82d3ca700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:03.269+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:04.870+0000 7fe82d3ca700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.156:6800/2588638081 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:04.870+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:06.709+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mgrmap(e 10) ==== 50002+0+0 (secure 0 0 0) 0x7fe81801baa0 con 0x7fe828109430 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:06.709+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 msgr2=0x7fe814040980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:06.709+0000 7fe81effd700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.711+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mgrmap(e 11) ==== 50129+0+0 (secure 0 0 0) 0x7fe81802dd10 con 0x7fe828109430 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.711+0000 7fe81effd700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 0x7fe814044330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.711+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fe818008e50 con 0x7fe814041f40 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.711+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 0x7fe814044330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.712+0000 7fe82d3ca700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 0x7fe814044330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe824003a10 tx=0x7fe8240092b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.713+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7fe818008e50 con 0x7fe814041f40 2024-05-26T11:53:08.486 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.721+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fe82806a680 con 0x7fe814041f40 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.721+0000 7fe81effd700 1 -- 172.21.15.156:0/3377249419 <== mgr.14150 v2:172.21.15.156:6800/3256012970 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fe82806a680 con 0x7fe814041f40 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 msgr2=0x7fe814044330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 0x7fe814044330 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fe824003a10 tx=0x7fe8240092b0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 msgr2=0x7fe82819eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe818004660 tx=0x7fe818011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 shutdown_connections 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe814041f40 0x7fe814044330 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe828109430 0x7fe82819eff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 --2- 172.21.15.156:0/3377249419 >> [v2:172.21.15.156:6800/2588638081,v1:172.21.15.156:6801/2588638081] conn(0x7fe81403e4c0 0x7fe814040980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 >> 172.21.15.156:0/3377249419 conn(0x7fe828102ba0 msgr2=0x7fe828195610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 shutdown_connections 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:07.722+0000 7fe82fe2f700 1 -- 172.21.15.156:0/3377249419 wait complete. 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:mgr epoch 9 is available 2024-05-26T11:53:08.487 INFO:teuthology.orchestra.run.smithi156.stdout:Generating a dashboard self-signed certificate... 2024-05-26T11:53:08.860 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:08 smithi156 ceph-mon[32025]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-26T11:53:08.860 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:08 smithi156 ceph-mon[32025]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-26T11:53:09.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:09 smithi156 ceph-mon[32025]: mgrmap e12: x(active, since 2s) 2024-05-26T11:53:09.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:09 smithi156 ceph-mon[32025]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.964+0000 7f3ccb6c4700 1 Processor -- start 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.964+0000 7f3ccb6c4700 1 -- start start 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3ccb6c4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3ccb6c4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc41087c0 con 0x7f3cc4107e60 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cc9460700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cc9460700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57644/0 (socket says 172.21.15.156:57644) 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cc9460700 1 -- 172.21.15.156:0/902245935 learned_addr learned my addr 172.21.15.156:0/902245935 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cc9460700 1 -- 172.21.15.156:0/902245935 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc4108fc0 con 0x7f3cc4107e60 2024-05-26T11:53:09.955 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cc9460700 1 --2- 172.21.15.156:0/902245935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00098a0 tx=0x7f3cc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f3f7e58662cf3b server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cbbfff700 1 -- 172.21.15.156:0/902245935 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0004160 con 0x7f3cc4107e60 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.965+0000 7f3cbbfff700 1 -- 172.21.15.156:0/902245935 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc0004300 con 0x7f3cc4107e60 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3cbbfff700 1 -- 172.21.15.156:0/902245935 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0011a20 con 0x7f3cc4107e60 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/902245935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 msgr2=0x7f3cc4108280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/902245935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3cc00098a0 tx=0x7f3cc0009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/902245935 shutdown_connections 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/902245935 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4108280 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/902245935 >> 172.21.15.156:0/902245935 conn(0x7f3cc4103420 msgr2=0x7f3cc4105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/902245935 shutdown_connections 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.966+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/902245935 wait complete. 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3ccb6c4700 1 Processor -- start 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3ccb6c4700 1 -- start start 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3ccb6c4700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:09.956 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3ccb6c4700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3cc410c400 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3cc9460700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3cc9460700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57656/0 (socket says 172.21.15.156:57656) 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.967+0000 7f3cc9460700 1 -- 172.21.15.156:0/1940516396 learned_addr learned my addr 172.21.15.156:0/1940516396 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3cc9460700 1 -- 172.21.15.156:0/1940516396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cc4193470 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3cc9460700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0004660 tx=0x7f3cc0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0003aa0 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cc4191660 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc001b8f0 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3cc0023900 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.968+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cc4191ba0 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.969+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f3cc0023b70 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.969+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cc410c580 con 0x7f3cc4107e60 2024-05-26T11:53:09.957 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.970+0000 7f3cba7fc700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 0x7f3cb00409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.970+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3cc0053ef0 con 0x7f3cc4107e60 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.970+0000 7f3cc8c5f700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 0x7f3cb00409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.970+0000 7f3cc8c5f700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 0x7f3cb00409e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:08.975+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3cc001f030 con 0x7f3cc4107e60 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.196+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f3cc4196cd0 con 0x7f3cb003e520 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.290+0000 7f3cba7fc700 1 -- 172.21.15.156:0/1940516396 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f3cc4196cd0 con 0x7f3cb003e520 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 msgr2=0x7f3cb00409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 0x7f3cb00409e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f3cb4006fd0 tx=0x7f3cb4006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 msgr2=0x7f3cc4192f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3cc0004660 tx=0x7f3cc0011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 shutdown_connections 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3cb003e520 0x7f3cb00409e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:09.958 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 --2- 172.21.15.156:0/1940516396 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cc4107e60 0x7f3cc4192f30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:09.959 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 >> 172.21.15.156:0/1940516396 conn(0x7f3cc4103420 msgr2=0x7f3cc4104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:09.959 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 shutdown_connections 2024-05-26T11:53:09.959 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:09.293+0000 7f3ccb6c4700 1 -- 172.21.15.156:0/1940516396 wait complete. 2024-05-26T11:53:09.959 INFO:teuthology.orchestra.run.smithi156.stdout:Creating initial admin user... 2024-05-26T11:53:11.471 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$nIleKc9TeQCkuMHkFc9U9Ord.9a3vO5fXKltJandNgdeflhV3nNUC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1716724390, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f1d838700 1 Processor -- start 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f1d838700 1 -- start start 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f1d838700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f1d838700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1807d6b0 con 0x7f0f18107810 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57660/0 (socket says 172.21.15.156:57660) 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.438+0000 7f0f16ffd700 1 -- 172.21.15.156:0/2164170458 learned_addr learned my addr 172.21.15.156:0/2164170458 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f16ffd700 1 -- 172.21.15.156:0/2164170458 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f18108170 con 0x7f0f18107810 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f16ffd700 1 --2- 172.21.15.156:0/2164170458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f080098a0 tx=0x7f0f08009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae55c27a47785fe server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f15ffb700 1 -- 172.21.15.156:0/2164170458 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f08004160 con 0x7f0f18107810 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f15ffb700 1 -- 172.21.15.156:0/2164170458 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f08004300 con 0x7f0f18107810 2024-05-26T11:53:11.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f15ffb700 1 -- 172.21.15.156:0/2164170458 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f08011a20 con 0x7f0f18107810 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f1d838700 1 -- 172.21.15.156:0/2164170458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 msgr2=0x7f0f18107c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.439+0000 7f0f1d838700 1 --2- 172.21.15.156:0/2164170458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0f080098a0 tx=0x7f0f08009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 -- 172.21.15.156:0/2164170458 shutdown_connections 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 --2- 172.21.15.156:0/2164170458 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f18107c30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 -- 172.21.15.156:0/2164170458 >> 172.21.15.156:0/2164170458 conn(0x7f0f18103430 msgr2=0x7f0f18105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 -- 172.21.15.156:0/2164170458 shutdown_connections 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 -- 172.21.15.156:0/2164170458 wait complete. 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 Processor -- start 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.440+0000 7f0f1d838700 1 -- start start 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1d838700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1d838700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f0f181091e0 con 0x7f0f18107810 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57674/0 (socket says 172.21.15.156:57674) 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f16ffd700 1 -- 172.21.15.156:0/3486628927 learned_addr learned my addr 172.21.15.156:0/3486628927 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:11.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f16ffd700 1 -- 172.21.15.156:0/3486628927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1819f5a0 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f16ffd700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f08004660 tx=0x7f0f08011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f08003aa0 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1819f830 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0f0801b8f0 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f08023900 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.441+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f181a2520 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.442+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f18064ba0 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.443+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f0f08023bc0 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.443+0000 7f0f1c836700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 0x7f0f00040980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.443+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0f08054250 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.443+0000 7f0f167fc700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 0x7f0f00040980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.443+0000 7f0f167fc700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 0x7f0f00040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006fd0 tx=0x7f0f0c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.448+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0f0801f030 con 0x7f0f18107810 2024-05-26T11:53:11.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.673+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f0f18198b40 con 0x7f0f0003e4c0 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.899+0000 7f0f1c836700 1 -- 172.21.15.156:0/3486628927 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f0f18198b40 con 0x7f0f0003e4c0 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.901+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 msgr2=0x7f0f00040980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.901+0000 7f0f1d838700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 0x7f0f00040980 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006fd0 tx=0x7f0f0c006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 msgr2=0x7f0f1819f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0f08004660 tx=0x7f0f08011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 shutdown_connections 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f0f0003e4c0 0x7f0f00040980 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 --2- 172.21.15.156:0/3486628927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0f18107810 0x7f0f1819f060 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 >> 172.21.15.156:0/3486628927 conn(0x7f0f18103430 msgr2=0x7f0f18195c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 shutdown_connections 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:10.902+0000 7f0f1d838700 1 -- 172.21.15.156:0/3486628927 wait complete. 2024-05-26T11:53:11.475 INFO:teuthology.orchestra.run.smithi156.stdout:Fetching dashboard port number... 2024-05-26T11:53:11.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:11.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:12.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:12 smithi156 ceph-mon[32025]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:12.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:12 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1278316996' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-26T11:53:13.199 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stdout 8443 2024-05-26T11:53:13.199 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.944+0000 7f3ea8547700 1 Processor -- start 2024-05-26T11:53:13.199 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.944+0000 7f3ea8547700 1 -- start start 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea8547700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea8547700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea0107c10 con 0x7f3ea0107220 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea62e3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea62e3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57690/0 (socket says 172.21.15.156:57690) 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea62e3700 1 -- 172.21.15.156:0/1526956771 learned_addr learned my addr 172.21.15.156:0/1526956771 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea62e3700 1 -- 172.21.15.156:0/1526956771 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea0107d90 con 0x7f3ea0107220 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.945+0000 7f3ea62e3700 1 --2- 172.21.15.156:0/1526956771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e940098a0 tx=0x7f3e94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd17d3731a4ba81 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea52e1700 1 -- 172.21.15.156:0/1526956771 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e94004160 con 0x7f3ea0107220 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea52e1700 1 -- 172.21.15.156:0/1526956771 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e94004300 con 0x7f3ea0107220 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea52e1700 1 -- 172.21.15.156:0/1526956771 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e94011a20 con 0x7f3ea0107220 2024-05-26T11:53:13.200 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 -- 172.21.15.156:0/1526956771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 msgr2=0x7f3ea0107640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1526956771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3e940098a0 tx=0x7f3e94009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 -- 172.21.15.156:0/1526956771 shutdown_connections 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1526956771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea0107640 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 -- 172.21.15.156:0/1526956771 >> 172.21.15.156:0/1526956771 conn(0x7f3ea0102ba0 msgr2=0x7f3ea0104fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.946+0000 7f3ea8547700 1 -- 172.21.15.156:0/1526956771 shutdown_connections 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea8547700 1 -- 172.21.15.156:0/1526956771 wait complete. 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea8547700 1 Processor -- start 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea8547700 1 -- start start 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea8547700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea8547700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3ea010d4d0 con 0x7f3ea0107220 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea62e3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea62e3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57696/0 (socket says 172.21.15.156:57696) 2024-05-26T11:53:13.201 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.947+0000 7f3ea62e3700 1 -- 172.21.15.156:0/1278316996 learned_addr learned my addr 172.21.15.156:0/1278316996 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3ea62e3700 1 -- 172.21.15.156:0/1278316996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ea019d410 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3ea62e3700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e94000c00 tx=0x7f3e94011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e94003aa0 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e9401b8f0 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ea019d6a0 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3e94023890 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.948+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ea01a0390 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.949+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 12) ==== 50235+0+0 (secure 0 0 0) 0x7f3e9402c3f0 con 0x7f3ea0107220 2024-05-26T11:53:13.202 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.949+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ea0064ba0 con 0x7f3ea0107220 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.949+0000 7f3e937fe700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 0x7f3e8c0409e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.949+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3e94053ed0 con 0x7f3ea0107220 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.949+0000 7f3ea5ae2700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 0x7f3e8c0409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.950+0000 7f3ea5ae2700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 0x7f3e8c0409e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c006fd0 tx=0x7f3e9c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:11.954+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3e9401f030 con 0x7f3ea0107220 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.163+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f3ea01a0980 con 0x7f3ea0107220 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.164+0000 7f3e937fe700 1 -- 172.21.15.156:0/1278316996 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f3e940202b0 con 0x7f3ea0107220 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 msgr2=0x7f3e8c0409e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 0x7f3e8c0409e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3e9c006fd0 tx=0x7f3e9c006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 msgr2=0x7f3ea019ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3e94000c00 tx=0x7f3e94011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 shutdown_connections 2024-05-26T11:53:13.203 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3e8c03e520 0x7f3e8c0409e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:13.204 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 --2- 172.21.15.156:0/1278316996 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3ea0107220 0x7f3ea019ced0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:13.204 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 >> 172.21.15.156:0/1278316996 conn(0x7f3ea0102ba0 msgr2=0x7f3ea0103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:13.204 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 shutdown_connections 2024-05-26T11:53:13.204 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:12.166+0000 7f3ea8547700 1 -- 172.21.15.156:0/1278316996 wait complete. 2024-05-26T11:53:13.208 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-26T11:53:13.208 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout disabled 2024-05-26T11:53:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-26T11:53:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:systemctl: stdout inactive 2024-05-26T11:53:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:firewalld.service is not enabled 2024-05-26T11:53:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-26T11:53:13.218 INFO:teuthology.orchestra.run.smithi156.stdout:Ceph Dashboard is now available at: 2024-05-26T11:53:13.218 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:13.218 INFO:teuthology.orchestra.run.smithi156.stdout: URL: https://smithi156.front.sepia.ceph.com:8443/ 2024-05-26T11:53:13.219 INFO:teuthology.orchestra.run.smithi156.stdout: User: admin 2024-05-26T11:53:13.219 INFO:teuthology.orchestra.run.smithi156.stdout: Password: nk1u76kww0 2024-05-26T11:53:13.219 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:13.219 INFO:teuthology.orchestra.run.smithi156.stdout:Saving cluster configuration to /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config directory 2024-05-26T11:53:13.219 INFO:teuthology.orchestra.run.smithi156.stdout:Enabling autotune for osd_memory_target 2024-05-26T11:53:14.464 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:14 smithi156 ceph-mon[32025]: mgrmap e13: x(active, since 6s) 2024-05-26T11:53:14.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.660+0000 7f377e79a700 1 Processor -- start 2024-05-26T11:53:14.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.660+0000 7f377e79a700 1 -- start start 2024-05-26T11:53:14.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.660+0000 7f377e79a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:14.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.660+0000 7f377e79a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3778109890 con 0x7f3778108f30 2024-05-26T11:53:14.472 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f3777fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f3777fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57706/0 (socket says 172.21.15.156:57706) 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f3777fff700 1 -- 172.21.15.156:0/2235136733 learned_addr learned my addr 172.21.15.156:0/2235136733 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f3777fff700 1 -- 172.21.15.156:0/2235136733 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377810a090 con 0x7f3778108f30 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f3777fff700 1 --2- 172.21.15.156:0/2235136733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f37680098a0 tx=0x7f3768009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220188e31d7e60cc server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f37777fe700 1 -- 172.21.15.156:0/2235136733 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3768004160 con 0x7f3778108f30 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f37777fe700 1 -- 172.21.15.156:0/2235136733 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3768004300 con 0x7f3778108f30 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.661+0000 7f37777fe700 1 -- 172.21.15.156:0/2235136733 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3768011a20 con 0x7f3778108f30 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 -- 172.21.15.156:0/2235136733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 msgr2=0x7f3778109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 --2- 172.21.15.156:0/2235136733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f37680098a0 tx=0x7f3768009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 -- 172.21.15.156:0/2235136733 shutdown_connections 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 --2- 172.21.15.156:0/2235136733 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f3778109350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 -- 172.21.15.156:0/2235136733 >> 172.21.15.156:0/2235136733 conn(0x7f3778104550 msgr2=0x7f3778106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:14.473 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 -- 172.21.15.156:0/2235136733 shutdown_connections 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.662+0000 7f377e79a700 1 -- 172.21.15.156:0/2235136733 wait complete. 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 Processor -- start 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 -- start start 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f377810d4d0 con 0x7f3778108f30 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3777fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3777fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:57714/0 (socket says 172.21.15.156:57714) 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3777fff700 1 -- 172.21.15.156:0/1395330727 learned_addr learned my addr 172.21.15.156:0/1395330727 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3777fff700 1 -- 172.21.15.156:0/1395330727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f377819d360 con 0x7f3778108f30 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3777fff700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3768004660 tx=0x7f3768011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3768003aa0 con 0x7f3778108f30 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f376801b8f0 con 0x7f3778108f30 2024-05-26T11:53:14.474 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3768023900 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f377819d5f0 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.663+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37781a02e0 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.664+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f376801baa0 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.664+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f377810d650 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.665+0000 7f3775ffb700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 0x7f3764040a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.665+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3768053f50 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.665+0000 7f376ffff700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 0x7f3764040a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.665+0000 7f376ffff700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 0x7f3764040a80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3760006fd0 tx=0x7f3760006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.670+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f376801f030 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.872+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f37781a0930 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.872+0000 7f3775ffb700 1 -- 172.21.15.156:0/1395330727 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f3768018350 con 0x7f3778108f30 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 msgr2=0x7f3764040a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 0x7f3764040a80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3760006fd0 tx=0x7f3760006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:14.475 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 msgr2=0x7f377819ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3768004660 tx=0x7f3768011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 shutdown_connections 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f376403e5c0 0x7f3764040a80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 --2- 172.21.15.156:0/1395330727 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3778108f30 0x7f377819ce20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 >> 172.21.15.156:0/1395330727 conn(0x7f3778104550 msgr2=0x7f3778104fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 shutdown_connections 2024-05-26T11:53:14.476 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:13.874+0000 7f377e79a700 1 -- 172.21.15.156:0/1395330727 wait complete. 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49bba75700 1 Processor -- start 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49bba75700 1 -- start start 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49bba75700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49bba75700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b41098e0 con 0x7f49b4108f80 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49b9811700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49b9811700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49448/0 (socket says 172.21.15.156:49448) 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.920+0000 7f49b9811700 1 -- 172.21.15.156:0/1047484258 learned_addr learned my addr 172.21.15.156:0/1047484258 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49b9811700 1 -- 172.21.15.156:0/1047484258 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b410a090 con 0x7f49b4108f80 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49b9811700 1 --2- 172.21.15.156:0/1047484258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49b00098a0 tx=0x7f49b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6c68d104f97b57a server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:15.820 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49b880f700 1 -- 172.21.15.156:0/1047484258 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b0004160 con 0x7f49b4108f80 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49b880f700 1 -- 172.21.15.156:0/1047484258 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b0004300 con 0x7f49b4108f80 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49b880f700 1 -- 172.21.15.156:0/1047484258 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b0011a20 con 0x7f49b4108f80 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49bba75700 1 -- 172.21.15.156:0/1047484258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 msgr2=0x7f49b41093a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.921+0000 7f49bba75700 1 --2- 172.21.15.156:0/1047484258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f49b00098a0 tx=0x7f49b0009bb0 comp rx=0 tx=0).stop 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- 172.21.15.156:0/1047484258 shutdown_connections 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 --2- 172.21.15.156:0/1047484258 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b41093a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- 172.21.15.156:0/1047484258 >> 172.21.15.156:0/1047484258 conn(0x7f49b41045a0 msgr2=0x7f49b41069c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- 172.21.15.156:0/1047484258 shutdown_connections 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- 172.21.15.156:0/1047484258 wait complete. 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 Processor -- start 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- start start 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:15.821 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49bba75700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f49b410d4d0 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.922+0000 7f49b9811700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49b9811700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49450/0 (socket says 172.21.15.156:49450) 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49b9811700 1 -- 172.21.15.156:0/3135041395 learned_addr learned my addr 172.21.15.156:0/3135041395 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49b9811700 1 -- 172.21.15.156:0/3135041395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b419d340 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49b9811700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49b00116e0 tx=0x7f49b0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b0003b20 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b419d5d0 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f49b001b8f0 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f49b00238c0 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.923+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b41a02c0 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.924+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f49b0023b70 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.924+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b410d650 con 0x7f49b4108f80 2024-05-26T11:53:15.822 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.924+0000 7f49aaffd700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 0x7f49a00409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.924+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f49b00542f0 con 0x7f49b4108f80 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.924+0000 7f49b9010700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 0x7f49a00409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.925+0000 7f49b9010700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 0x7f49a00409d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f49a4006fd0 tx=0x7f49a4006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:14.929+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f49b001f030 con 0x7f49b4108f80 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.244+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f49b41a0870 con 0x7f49b4108f80 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.247+0000 7f49aaffd700 1 -- 172.21.15.156:0/3135041395 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v26)=0 set mgr/dashboard/cluster/status v26) ==== 153+0+0 (secure 0 0 0) 0x7f49b0012040 con 0x7f49b4108f80 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 msgr2=0x7f49a00409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 0x7f49a00409d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f49a4006fd0 tx=0x7f49a4006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 msgr2=0x7f49b419ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f49b00116e0 tx=0x7f49b0011aa0 comp rx=0 tx=0).stop 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 shutdown_connections 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49a003e510 0x7f49a00409d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:15.823 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 --2- 172.21.15.156:0/3135041395 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f49b4108f80 0x7f49b419ce00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.249+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 >> 172.21.15.156:0/3135041395 conn(0x7f49b41045a0 msgr2=0x7f49b4105040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.250+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 shutdown_connections 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr 2024-05-26T11:53:15.250+0000 7f49bba75700 1 -- 172.21.15.156:0/3135041395 wait complete. 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:Or, if you are only running a single cluster on this host: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-26T11:53:15.824 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout: ceph telemetry on 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout:For more information see: 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:15.825 INFO:teuthology.orchestra.run.smithi156.stdout:Bootstrap complete. 2024-05-26T11:53:15.856 INFO:tasks.cephadm:Fetching config... 2024-05-26T11:53:15.857 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:53:15.857 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-26T11:53:15.872 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-26T11:53:15.873 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:53:15.873 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-26T11:53:15.929 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-26T11:53:15.929 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:53:15.929 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-26T11:53:15.994 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-26T11:53:15.994 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:53:15.995 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-26T11:53:16.050 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-26T11:53:16.050 DEBUG:teuthology.orchestra.run.smithi156:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-26T11:53:16.123 INFO:teuthology.orchestra.run.smithi156.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:16.135 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-26T11:53:16.174 INFO:teuthology.orchestra.run.smithi186.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:16.186 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-26T11:53:16.224 INFO:teuthology.orchestra.run.smithi192.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCmrFkFBeEwqsPoeqLjvQFmpow1Pi8tEvPx8gkQV8v7gYiBU0BBSEdOG+aXZ+jVEHRfewGBHhlZ69MF2OB1CH3sq5gdCGg5027YLkZ+XLyfxaHFF8pjPdh5ogBCttxlNLMmQBY17faPU2ZVNAEAj5gOR+5CmX25ospxzome4EOPa/zLa8Q+00uc1AOuVmMl7k5dLxjr+GBDCfT768uHp/g4BuQ2W4judNHXOM5BtH/jkR/9plQFQLGajwwaZ6rhDFhBvuihMot1CdSzEjGnVZ8qCVmJKz4VaN/ptUad8PmOY0hlqBuw6nL6Et/tg3pFrTSCszYUFgkQqJnYOErUOIbaypNzTU3XRaW12dL1EfFdgaYlWJpqA5LWEmonBvEGGTGk4r+BDxu9D+V7bxFZ+rg4ZMiK900OQDfBgp17Vg6nRUdNHqrXfxVKafFeRqH5Y2Ve1exZ8eHboMdRRGJKAATJ3d/Sh8Zw0JtJ6y0pAp3bM5RhcLazTbjp7zLv6vxef3k= ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:16.236 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-26T11:53:16.366 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:16 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3135041395' entity='client.admin' 2024-05-26T11:53:16.479 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:18.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 -- 172.21.15.156:0/508452711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 msgr2=0x7f80bc105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:18.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 --2- 172.21.15.156:0/508452711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc105e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80b8009910 tx=0x7f80b8009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:18.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 -- 172.21.15.156:0/508452711 shutdown_connections 2024-05-26T11:53:18.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 --2- 172.21.15.156:0/508452711 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc105e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:18.621 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 -- 172.21.15.156:0/508452711 >> 172.21.15.156:0/508452711 conn(0x7f80bc0fd450 msgr2=0x7f80bc0ff890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 -- 172.21.15.156:0/508452711 shutdown_connections 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.619+0000 7f80c3880700 1 -- 172.21.15.156:0/508452711 wait complete. 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c3880700 1 Processor -- start 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c3880700 1 -- start start 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c3880700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c3880700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f80bc108e90 con 0x7f80bc103a40 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c161c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c161c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49454/0 (socket says 172.21.15.156:49454) 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c161c700 1 -- 172.21.15.156:0/2157968675 learned_addr learned my addr 172.21.15.156:0/2157968675 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c161c700 1 -- 172.21.15.156:0/2157968675 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80bc198ae0 con 0x7f80bc103a40 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.620+0000 7f80c161c700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80b8005b40 tx=0x7f80b8004f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.621+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80b8027070 con 0x7f80bc103a40 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.621+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80b800be10 con 0x7f80bc103a40 2024-05-26T11:53:18.622 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.621+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80bc198dd0 con 0x7f80bc103a40 2024-05-26T11:53:18.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.621+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80b8022d80 con 0x7f80bc103a40 2024-05-26T11:53:18.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.621+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80bc19ba60 con 0x7f80bc103a40 2024-05-26T11:53:18.623 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f80b802a070 con 0x7f80bc103a40 2024-05-26T11:53:18.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80bc109610 con 0x7f80bc103a40 2024-05-26T11:53:18.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80b27fc700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 0x7f80a80409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:18.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f80b801e030 con 0x7f80bc103a40 2024-05-26T11:53:18.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80c0e1b700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 0x7f80a80409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:18.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.622+0000 7f80c0e1b700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 0x7f80a80409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80ac006fd0 tx=0x7f80ac006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:18.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.627+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f80b8013070 con 0x7f80bc103a40 2024-05-26T11:53:18.832 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.832+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f80bc02e6e0 con 0x7f80bc103a40 2024-05-26T11:53:18.834 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.834+0000 7f80b27fc700 1 -- 172.21.15.156:0/2157968675 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f80b80310c0 con 0x7f80bc103a40 2024-05-26T11:53:18.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 msgr2=0x7f80a80409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:18.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 0x7f80a80409d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f80ac006fd0 tx=0x7f80ac006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:18.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 msgr2=0x7f80bc1985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f80b8005b40 tx=0x7f80b8004f80 comp rx=0 tx=0).stop 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 shutdown_connections 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80a803e510 0x7f80a80409d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 --2- 172.21.15.156:0/2157968675 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f80bc103a40 0x7f80bc1985a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 >> 172.21.15.156:0/2157968675 conn(0x7f80bc0fd450 msgr2=0x7f80bc0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 shutdown_connections 2024-05-26T11:53:18.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:18.837+0000 7f80c3880700 1 -- 172.21.15.156:0/2157968675 wait complete. 2024-05-26T11:53:19.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:19.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:19.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi156", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:53:19.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:19.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:19.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:19.385 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-26T11:53:19.386 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-26T11:53:19.621 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:20.019 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:19 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2157968675' entity='client.admin' 2024-05-26T11:53:20.019 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:19 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:20.020 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:19 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:21.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 -- 172.21.15.156:0/3114489139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 msgr2=0x7f856c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:21.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 --2- 172.21.15.156:0/3114489139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c102330 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f855c00b0a0 tx=0x7f855c00b3b0 comp rx=0 tx=0).stop 2024-05-26T11:53:21.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 -- 172.21.15.156:0/3114489139 shutdown_connections 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 --2- 172.21.15.156:0/3114489139 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c102330 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 -- 172.21.15.156:0/3114489139 >> 172.21.15.156:0/3114489139 conn(0x7f856c0fd4b0 msgr2=0x7f856c0ff8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 -- 172.21.15.156:0/3114489139 shutdown_connections 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.145+0000 7f8574316700 1 -- 172.21.15.156:0/3114489139 wait complete. 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 Processor -- start 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 -- start start 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f856c107d50 con 0x7f856c101f10 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f85720b2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f85720b2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49466/0 (socket says 172.21.15.156:49466) 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f85720b2700 1 -- 172.21.15.156:0/1730581577 learned_addr learned my addr 172.21.15.156:0/1730581577 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f85720b2700 1 -- 172.21.15.156:0/1730581577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c1979a0 con 0x7f856c101f10 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f85720b2700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f855c015040 tx=0x7f855c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:21.148 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f855c021020 con 0x7f856c101f10 2024-05-26T11:53:21.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f855c00bdb0 con 0x7f856c101f10 2024-05-26T11:53:21.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f856c197c30 con 0x7f856c101f10 2024-05-26T11:53:21.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f855c017040 con 0x7f856c101f10 2024-05-26T11:53:21.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.146+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f856c19a920 con 0x7f856c101f10 2024-05-26T11:53:21.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.147+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f855c024020 con 0x7f856c101f10 2024-05-26T11:53:21.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.147+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f856c1084d0 con 0x7f856c101f10 2024-05-26T11:53:21.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.148+0000 7f856b7fe700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 0x7f8554040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:21.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.148+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f855c00f050 con 0x7f856c101f10 2024-05-26T11:53:21.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.148+0000 7f85718b1700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 0x7f8554040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:21.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.148+0000 7f85718b1700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 0x7f8554040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8560006fd0 tx=0x7f8560006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:21.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.153+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f855c02b020 con 0x7f856c101f10 2024-05-26T11:53:21.358 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.357+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f856c02e5d0 con 0x7f855403e570 2024-05-26T11:53:21.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.361+0000 7f856b7fe700 1 -- 172.21.15.156:0/1730581577 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f856c02e5d0 con 0x7f855403e570 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 msgr2=0x7f8554040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 0x7f8554040a30 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f8560006fd0 tx=0x7f8560006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 msgr2=0x7f856c197460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f855c015040 tx=0x7f855c003ce0 comp rx=0 tx=0).stop 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 shutdown_connections 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f855403e570 0x7f8554040a30 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 --2- 172.21.15.156:0/1730581577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f856c101f10 0x7f856c197460 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 >> 172.21.15.156:0/1730581577 conn(0x7f856c0fd4b0 msgr2=0x7f856c0fdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 shutdown_connections 2024-05-26T11:53:21.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:21.363+0000 7f8574316700 1 -- 172.21.15.156:0/1730581577 wait complete. 2024-05-26T11:53:22.037 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi186 2024-05-26T11:53:22.038 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:53:22.038 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.conf 2024-05-26T11:53:22.058 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:53:22.058 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:22.115 INFO:tasks.cephadm:Adding host smithi186 to orchestrator... 2024-05-26T11:53:22.116 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch host add smithi186 2024-05-26T11:53:22.372 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: Updating smithi156:/etc/ceph/ceph.conf 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:22.618 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:23.464 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:23 smithi156 ceph-mon[32025]: Updating smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:23.464 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:23 smithi156 ceph-mon[32025]: Updating smithi156:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:23.464 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:23 smithi156 ceph-mon[32025]: Updating smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:53:23.914 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 -- 172.21.15.156:0/162096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 msgr2=0x7f3a10102460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 --2- 172.21.15.156:0/162096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10102460 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00b0a0 tx=0x7f3a0c00b3b0 comp rx=0 tx=0).stop 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 -- 172.21.15.156:0/162096 shutdown_connections 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 --2- 172.21.15.156:0/162096 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10102460 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 -- 172.21.15.156:0/162096 >> 172.21.15.156:0/162096 conn(0x7f3a10078b10 msgr2=0x7f3a100ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 -- 172.21.15.156:0/162096 shutdown_connections 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.913+0000 7f3a1812e700 1 -- 172.21.15.156:0/162096 wait complete. 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a1812e700 1 Processor -- start 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a1812e700 1 -- start start 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a1812e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a1812e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1010c300 con 0x7f3a10102080 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a15eca700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a15eca700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:49480/0 (socket says 172.21.15.156:49480) 2024-05-26T11:53:23.915 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a15eca700 1 -- 172.21.15.156:0/1765832673 learned_addr learned my addr 172.21.15.156:0/1765832673 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a15eca700 1 -- 172.21.15.156:0/1765832673 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a101059c0 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.914+0000 7f3a15eca700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c006130 tx=0x7f3a0c0097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.915+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a0c021020 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.915+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a0c0079b0 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.915+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a10104060 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.915+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3a0c017040 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.915+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a10104520 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a10103560 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f3a0c024020 con 0x7f3a10102080 2024-05-26T11:53:23.917 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a06ffd700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 0x7f39fc040a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:23.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3a0c00f050 con 0x7f3a10102080 2024-05-26T11:53:23.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a156c9700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 0x7f39fc040a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:23.918 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.916+0000 7f3a156c9700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 0x7f39fc040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006fd0 tx=0x7f3a00006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:23.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:23.921+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3a0c02b020 con 0x7f3a10102080 2024-05-26T11:53:24.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:24.131+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}) -- 0x7f3a10105000 con 0x7f39fc03e570 2024-05-26T11:53:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:24 smithi156 ceph-mon[32025]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi186", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:25 smithi156 ceph-mon[32025]: Deploying cephadm binary to smithi186 2024-05-26T11:53:28.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:27 smithi156 ceph-mon[32025]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:28.241 INFO:teuthology.orchestra.run.smithi156.stdout:Added host 'smithi186' with addr '172.21.15.186' 2024-05-26T11:53:28.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.239+0000 7f3a06ffd700 1 -- 172.21.15.156:0/1765832673 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f3a10105000 con 0x7f39fc03e570 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 msgr2=0x7f39fc040a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 0x7f39fc040a30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3a00006fd0 tx=0x7f3a00006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 msgr2=0x7f3a10103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c006130 tx=0x7f3a0c0097d0 comp rx=0 tx=0).stop 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 shutdown_connections 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f39fc03e570 0x7f39fc040a30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 --2- 172.21.15.156:0/1765832673 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3a10102080 0x7f3a10103b20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:28.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 >> 172.21.15.156:0/1765832673 conn(0x7f3a10078b10 msgr2=0x7f3a1006a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:28.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 shutdown_connections 2024-05-26T11:53:28.244 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:28.242+0000 7f3a1812e700 1 -- 172.21.15.156:0/1765832673 wait complete. 2024-05-26T11:53:28.849 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-26T11:53:29.085 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:29.531 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:29.531 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:29 smithi156 ceph-mon[32025]: Added host smithi186 2024-05-26T11:53:29.531 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:29.531 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:30.420 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:30 smithi156 ceph-mon[32025]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:30.421 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:30 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 -- 172.21.15.156:0/1050497817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 msgr2=0x7fd24c108340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1050497817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c108340 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd23c009910 tx=0x7fd23c009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 -- 172.21.15.156:0/1050497817 shutdown_connections 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1050497817 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c108340 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 -- 172.21.15.156:0/1050497817 >> 172.21.15.156:0/1050497817 conn(0x7fd24c0fd0c0 msgr2=0x7fd24c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 -- 172.21.15.156:0/1050497817 shutdown_connections 2024-05-26T11:53:30.624 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.623+0000 7fd250a1a700 1 -- 172.21.15.156:0/1050497817 wait complete. 2024-05-26T11:53:30.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd250a1a700 1 Processor -- start 2024-05-26T11:53:30.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd250a1a700 1 -- start start 2024-05-26T11:53:30.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd250a1a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:30.626 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd250a1a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd24c10d570 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd24a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd24a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53340/0 (socket says 172.21.15.156:53340) 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd24a59c700 1 -- 172.21.15.156:0/1458176088 learned_addr learned my addr 172.21.15.156:0/1458176088 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.624+0000 7fd24a59c700 1 -- 172.21.15.156:0/1458176088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24c104450 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd24a59c700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0056d0 tx=0x7fd23c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd23c026070 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd24c102aa0 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd24c102f60 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd23c00b750 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.625+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd23c004840 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.626+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd22c005330 con 0x7fd24c107f60 2024-05-26T11:53:30.627 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.626+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fd23c029070 con 0x7fd24c107f60 2024-05-26T11:53:30.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.626+0000 7fd23b7fe700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 0x7fd2340409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:30.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.626+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd23c01e070 con 0x7fd24c107f60 2024-05-26T11:53:30.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.626+0000 7fd249d9b700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 0x7fd2340409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:30.628 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.627+0000 7fd249d9b700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 0x7fd2340409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd240006fd0 tx=0x7fd240006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:30.632 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.631+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd23c013070 con 0x7fd24c107f60 2024-05-26T11:53:30.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.839+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd22c000bc0 con 0x7fd23403e510 2024-05-26T11:53:30.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.840+0000 7fd23b7fe700 1 -- 172.21.15.156:0/1458176088 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+161 (secure 0 0 0) 0x7fd22c000bc0 con 0x7fd23403e510 2024-05-26T11:53:30.840 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:30.841 INFO:teuthology.orchestra.run.smithi156.stdout:[{"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}] 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 msgr2=0x7fd2340409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 0x7fd2340409d0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd240006fd0 tx=0x7fd240006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 msgr2=0x7fd24c102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd23c0056d0 tx=0x7fd23c0057b0 comp rx=0 tx=0).stop 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 shutdown_connections 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd23403e510 0x7fd2340409d0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 --2- 172.21.15.156:0/1458176088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd24c107f60 0x7fd24c102560 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 >> 172.21.15.156:0/1458176088 conn(0x7fd24c0fd0c0 msgr2=0x7fd24c0fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 shutdown_connections 2024-05-26T11:53:30.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:30.842+0000 7fd250a1a700 1 -- 172.21.15.156:0/1458176088 wait complete. 2024-05-26T11:53:31.437 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi192 2024-05-26T11:53:31.437 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:53:31.437 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.conf 2024-05-26T11:53:31.456 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:53:31.456 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:31.513 INFO:tasks.cephadm:Adding host smithi192 to orchestrator... 2024-05-26T11:53:31.513 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch host add smithi192 2024-05-26T11:53:31.753 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:32.168 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:31 smithi156 ceph-mon[32025]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:32.168 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:31 smithi156 ceph-mon[32025]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T11:53:32.169 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.184+0000 7fa4a015f700 1 -- 172.21.15.156:0/1906106728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 msgr2=0x7fa498109340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.184+0000 7fa4a015f700 1 --2- 172.21.15.156:0/1906106728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498109340 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa494009910 tx=0x7fa494009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.184+0000 7fa4a015f700 1 -- 172.21.15.156:0/1906106728 shutdown_connections 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.184+0000 7fa4a015f700 1 --2- 172.21.15.156:0/1906106728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498109340 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.184+0000 7fa4a015f700 1 -- 172.21.15.156:0/1906106728 >> 172.21.15.156:0/1906106728 conn(0x7fa498079440 msgr2=0x7fa498079850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 -- 172.21.15.156:0/1906106728 shutdown_connections 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 -- 172.21.15.156:0/1906106728 wait complete. 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 Processor -- start 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 -- start start 2024-05-26T11:53:33.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa4a015f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa49810d430 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.185+0000 7fa49defb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa49defb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:53348/0 (socket says 172.21.15.156:53348) 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa49defb700 1 -- 172.21.15.156:0/2178580783 learned_addr learned my addr 172.21.15.156:0/2178580783 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa49defb700 1 -- 172.21.15.156:0/2178580783 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa498104700 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa49defb700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4940056d0 tx=0x7fa4940057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa494026070 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa498102db0 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa49400bc00 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa494003920 con 0x7fa498108f60 2024-05-26T11:53:33.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.186+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa498103270 con 0x7fa498108f60 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.187+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fa494029070 con 0x7fa498108f60 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.187+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa49806b0d0 con 0x7fa498108f60 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.187+0000 7fa48effd700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 0x7fa4840409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.187+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa49401e070 con 0x7fa498108f60 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.187+0000 7fa49d6fa700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 0x7fa4840409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:33.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.188+0000 7fa49d6fa700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 0x7fa4840409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa488006fd0 tx=0x7fa488006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:33.193 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.192+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa494013070 con 0x7fa498108f60 2024-05-26T11:53:33.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:33.399+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}) -- 0x7fa498103db0 con 0x7fa48403e510 2024-05-26T11:53:34.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:33 smithi156 ceph-mon[32025]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:34.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:33 smithi156 ceph-mon[32025]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:35.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: Deploying cephadm binary to smithi192 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:53:35.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:35 smithi156 ceph-mon[32025]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:36.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:36 smithi156 ceph-mon[32025]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:37.473 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.472+0000 7fa48effd700 1 -- 172.21.15.156:0/2178580783 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fa498103db0 con 0x7fa48403e510 2024-05-26T11:53:37.473 INFO:teuthology.orchestra.run.smithi156.stdout:Added host 'smithi192' with addr '172.21.15.192' 2024-05-26T11:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 msgr2=0x7fa4840409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 0x7fa4840409d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa488006fd0 tx=0x7fa488006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 msgr2=0x7fa498102810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa4940056d0 tx=0x7fa4940057b0 comp rx=0 tx=0).stop 2024-05-26T11:53:37.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 shutdown_connections 2024-05-26T11:53:37.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa48403e510 0x7fa4840409d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:37.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 --2- 172.21.15.156:0/2178580783 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa498108f60 0x7fa498102810 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:37.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 >> 172.21.15.156:0/2178580783 conn(0x7fa498079440 msgr2=0x7fa4980fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:37.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 shutdown_connections 2024-05-26T11:53:37.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:37.475+0000 7fa4a015f700 1 -- 172.21.15.156:0/2178580783 wait complete. 2024-05-26T11:53:38.262 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch host ls --format=json 2024-05-26T11:53:38.497 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: Added host smithi192 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:39.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:39.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:39 smithi156 ceph-mon[32025]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:40.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 -- 172.21.15.156:0/208343794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 msgr2=0x7f1834102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 --2- 172.21.15.156:0/208343794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f1834102490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1824009910 tx=0x7f1824009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 -- 172.21.15.156:0/208343794 shutdown_connections 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 --2- 172.21.15.156:0/208343794 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f1834102490 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 -- 172.21.15.156:0/208343794 >> 172.21.15.156:0/208343794 conn(0x7f18340fd980 msgr2=0x7f18340ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 -- 172.21.15.156:0/208343794 shutdown_connections 2024-05-26T11:53:40.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 -- 172.21.15.156:0/208343794 wait complete. 2024-05-26T11:53:40.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.007+0000 7f1839d20700 1 Processor -- start 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f1839d20700 1 -- start start 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f1839d20700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f1839d20700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f183410c330 con 0x7f18341020b0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f18337fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f18337fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43792/0 (socket says 172.21.15.156:43792) 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f18337fe700 1 -- 172.21.15.156:0/1797659237 learned_addr learned my addr 172.21.15.156:0/1797659237 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f18337fe700 1 -- 172.21.15.156:0/1797659237 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183419eb80 con 0x7f18341020b0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.008+0000 7f18337fe700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18240056d0 tx=0x7f18240057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1824026070 con 0x7f18341020b0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183419ee70 con 0x7f18341020b0 2024-05-26T11:53:40.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f182400bba0 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1824003920 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18341987c0 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.009+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f1824029070 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.010+0000 7f1830ff9700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 0x7f181c0409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.010+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1834103590 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.010+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f182401e070 con 0x7f18341020b0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.010+0000 7f1832ffd700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 0x7f181c0409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:40.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.010+0000 7f1832ffd700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 0x7f181c0409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1828006fd0 tx=0x7f1828006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:40.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.015+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1824030070 con 0x7f18341020b0 2024-05-26T11:53:40.227 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.226+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1834199cf0 con 0x7f181c03e510 2024-05-26T11:53:40.228 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.227+0000 7f1830ff9700 1 -- 172.21.15.156:0/1797659237 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+241 (secure 0 0 0) 0x7f1834199cf0 con 0x7f181c03e510 2024-05-26T11:53:40.228 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:53:40.228 INFO:teuthology.orchestra.run.smithi156.stdout:[{"addr": "172.21.15.156", "hostname": "smithi156", "labels": [], "status": ""}, {"addr": "172.21.15.186", "hostname": "smithi186", "labels": [], "status": ""}, {"addr": "172.21.15.192", "hostname": "smithi192", "labels": [], "status": ""}] 2024-05-26T11:53:40.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 msgr2=0x7f181c0409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:40.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 0x7f181c0409d0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1828006fd0 tx=0x7f1828006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:40.230 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 msgr2=0x7f183419e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f18240056d0 tx=0x7f18240057b0 comp rx=0 tx=0).stop 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 shutdown_connections 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f181c03e510 0x7f181c0409d0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 --2- 172.21.15.156:0/1797659237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f18341020b0 0x7f183419e640 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.229+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 >> 172.21.15.156:0/1797659237 conn(0x7f18340fd980 msgr2=0x7f183406b380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.230+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 shutdown_connections 2024-05-26T11:53:40.231 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:40.230+0000 7f1839d20700 1 -- 172.21.15.156:0/1797659237 wait complete. 2024-05-26T11:53:40.829 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-26T11:53:40.829 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd crush tunables default 2024-05-26T11:53:41.065 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:53:41.377 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:41.378 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:41 smithi156 ceph-mon[32025]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T11:53:42.204 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:42 smithi156 ceph-mon[32025]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 -- 172.21.15.156:0/3892343912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 msgr2=0x7fbc44108320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/3892343912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc44108320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbc34009910 tx=0x7fbc34009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 -- 172.21.15.156:0/3892343912 shutdown_connections 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/3892343912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc44108320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 -- 172.21.15.156:0/3892343912 >> 172.21.15.156:0/3892343912 conn(0x7fbc44078530 msgr2=0x7fbc44078940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 -- 172.21.15.156:0/3892343912 shutdown_connections 2024-05-26T11:53:42.391 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.387+0000 7fbc4a13e700 1 -- 172.21.15.156:0/3892343912 wait complete. 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc4a13e700 1 Processor -- start 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc4a13e700 1 -- start start 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc4a13e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc4a13e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc4410d550 con 0x7fbc44107f40 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc437fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc437fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43818/0 (socket says 172.21.15.156:43818) 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.388+0000 7fbc437fe700 1 -- 172.21.15.156:0/878272529 learned_addr learned my addr 172.21.15.156:0/878272529 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc437fe700 1 -- 172.21.15.156:0/878272529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc4419ec10 con 0x7fbc44107f40 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc437fe700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc340056d0 tx=0x7fbc340057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc34026070 con 0x7fbc44107f40 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc4419eea0 con 0x7fbc44107f40 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc3400bc00 con 0x7fbc44107f40 2024-05-26T11:53:42.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbc34003920 con 0x7fbc44107f40 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.389+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc441988a0 con 0x7fbc44107f40 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fbc34029070 con 0x7fbc44107f40 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc44104850 con 0x7fbc44107f40 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc417fa700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 0x7fbc300409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbc3401e070 con 0x7fbc44107f40 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc3bfff700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 0x7fbc300409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:42.393 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.390+0000 7fbc3bfff700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 0x7fbc300409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c006fd0 tx=0x7fbc2c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:42.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.395+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbc34013070 con 0x7fbc44107f40 2024-05-26T11:53:42.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:42.604+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fbc44064ba0 con 0x7fbc44107f40 2024-05-26T11:53:43.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.124+0000 7fbc417fa700 1 -- 172.21.15.156:0/878272529 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fbc34030090 con 0x7fbc44107f40 2024-05-26T11:53:43.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 msgr2=0x7fbc300409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 0x7fbc300409d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c006fd0 tx=0x7fbc2c006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 msgr2=0x7fbc4419e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbc340056d0 tx=0x7fbc340057b0 comp rx=0 tx=0).stop 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 shutdown_connections 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc3003e510 0x7fbc300409d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 --2- 172.21.15.156:0/878272529 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc44107f40 0x7fbc4419e6d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 >> 172.21.15.156:0/878272529 conn(0x7fbc44078530 msgr2=0x7fbc44104b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 shutdown_connections 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:53:43.128+0000 7fbc4a13e700 1 -- 172.21.15.156:0/878272529 wait complete. 2024-05-26T11:53:43.129 INFO:teuthology.orchestra.run.smithi156.stderr:adjusted tunables profile to default 2024-05-26T11:53:43.496 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:43 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-26T11:53:43.838 INFO:tasks.cephadm:Adding mon.a on smithi156 2024-05-26T11:53:43.839 INFO:tasks.cephadm:Adding mon.b on smithi186 2024-05-26T11:53:43.839 INFO:tasks.cephadm:Adding mon.c on smithi192 2024-05-26T11:53:43.839 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch apply mon '3;smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c' 2024-05-26T11:53:44.075 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:44.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:44 smithi156 ceph-mon[32025]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.607+0000 7fef537bd700 1 -- 172.21.15.192:0/3312074489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 msgr2=0x7fef4c102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.607+0000 7fef537bd700 1 --2- 172.21.15.192:0/3312074489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c102490 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fef4800b0a0 tx=0x7fef4800b3b0 comp rx=0 tx=0).stop 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.607+0000 7fef537bd700 1 -- 172.21.15.192:0/3312074489 shutdown_connections 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.607+0000 7fef537bd700 1 --2- 172.21.15.192:0/3312074489 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c102490 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.607+0000 7fef537bd700 1 -- 172.21.15.192:0/3312074489 >> 172.21.15.192:0/3312074489 conn(0x7fef4c0fd980 msgr2=0x7fef4c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 -- 172.21.15.192:0/3312074489 shutdown_connections 2024-05-26T11:53:45.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 -- 172.21.15.192:0/3312074489 wait complete. 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 Processor -- start 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 -- start start 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.608+0000 7fef537bd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef4c10c330 con 0x7fef4c1020b0 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef51559700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef51559700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.192:43796/0 (socket says 172.21.15.192:43796) 2024-05-26T11:53:45.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef51559700 1 -- 172.21.15.192:0/655218486 learned_addr learned my addr 172.21.15.192:0/655218486 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:53:45.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef51559700 1 -- 172.21.15.192:0/655218486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef4c19ebb0 con 0x7fef4c1020b0 2024-05-26T11:53:45.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef51559700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef48006130 tx=0x7fef480097d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:45.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef48021020 con 0x7fef4c1020b0 2024-05-26T11:53:45.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef480079b0 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef4c19ee40 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fef48017040 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.609+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef4c198740 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.610+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4c103590 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.610+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fef48024020 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.610+0000 7fef427fc700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 0x7fef380409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.610+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fef4800f050 con 0x7fef4c1020b0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.611+0000 7fef50d58700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 0x7fef380409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:45.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.611+0000 7fef50d58700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 0x7fef380409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fef3c006fd0 tx=0x7fef3c006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:45.616 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.616+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fef4802b020 con 0x7fef4c1020b0 2024-05-26T11:53:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:53:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:45 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:45.826 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.826+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}) -- 0x7fef4c199d30 con 0x7fef3803e510 2024-05-26T11:53:45.833 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.833+0000 7fef427fc700 1 -- 172.21.15.192:0/655218486 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fef4c199d30 con 0x7fef3803e510 2024-05-26T11:53:45.833 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mon update... 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 msgr2=0x7fef380409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 0x7fef380409d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fef3c006fd0 tx=0x7fef3c006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 msgr2=0x7fef4c19e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fef48006130 tx=0x7fef480097d0 comp rx=0 tx=0).stop 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 shutdown_connections 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef3803e510 0x7fef380409d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 --2- 172.21.15.192:0/655218486 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef4c1020b0 0x7fef4c19e670 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 >> 172.21.15.192:0/655218486 conn(0x7fef4c0fd980 msgr2=0x7fef4c06b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 shutdown_connections 2024-05-26T11:53:45.836 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:45.835+0000 7fef537bd700 1 -- 172.21.15.192:0/655218486 wait complete. 2024-05-26T11:53:46.415 DEBUG:teuthology.orchestra.run.smithi186:mon.b> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.b.service 2024-05-26T11:53:46.418 DEBUG:teuthology.orchestra.run.smithi192:mon.c> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.c.service 2024-05-26T11:53:46.421 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-26T11:53:46.421 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:53:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:47.193 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:47.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:47 smithi156 ceph-mon[32025]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:47.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:47 smithi156 ceph-mon[32025]: Saving service mon spec with placement smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c;count:3 2024-05-26T11:53:47.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:47 smithi156 ceph-mon[32025]: Deploying daemon mon.c on smithi192 2024-05-26T11:53:48.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:48 smithi156 ceph-mon[32025]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:50.530 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.528+0000 7f35526fb700 1 -- 172.21.15.192:0/2441696056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 msgr2=0x7f354c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.528+0000 7f35526fb700 1 --2- 172.21.15.192:0/2441696056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c1045b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f353c009910 tx=0x7f353c009c20 comp rx=0 tx=0).stop 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 -- 172.21.15.192:0/2441696056 shutdown_connections 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 --2- 172.21.15.192:0/2441696056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c1045b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 -- 172.21.15.192:0/2441696056 >> 172.21.15.192:0/2441696056 conn(0x7f354c0fd980 msgr2=0x7f354c0ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 -- 172.21.15.192:0/2441696056 shutdown_connections 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 -- 172.21.15.192:0/2441696056 wait complete. 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 Processor -- start 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.529+0000 7f35526fb700 1 -- start start 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35526fb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35526fb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f354c10d490 con 0x7f354c1041d0 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f354bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f354bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.192:43822/0 (socket says 172.21.15.192:43822) 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f354bfff700 1 -- 172.21.15.192:0/110336255 learned_addr learned my addr 172.21.15.192:0/110336255 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:53:50.531 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f354bfff700 1 -- 172.21.15.192:0/110336255 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354c19ebd0 con 0x7f354c1041d0 2024-05-26T11:53:50.532 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f354bfff700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f353c0056d0 tx=0x7f353c0057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c026070 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f354c19ee60 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f353c004370 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.530+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f354c198810 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.531+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f353c0039b0 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.531+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3538005330 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.531+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f353c029070 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.532+0000 7f35497fa700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 0x7f35340409d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.532+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f353c01e070 con 0x7f354c1041d0 2024-05-26T11:53:50.533 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.532+0000 7f354b7fe700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 0x7f35340409d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:50.534 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.532+0000 7f354b7fe700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 0x7f35340409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3540006fd0 tx=0x7f3540006e40 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:50.537 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.537+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f353c013070 con 0x7f354c1041d0 2024-05-26T11:53:50.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:50 smithi156 ceph-mon[32025]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:50.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.854+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f3538005620 con 0x7f354c1041d0 2024-05-26T11:53:50.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.855+0000 7f35497fa700 1 -- 172.21.15.192:0/110336255 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+749 (secure 0 0 0) 0x7f353c030300 con 0x7f354c1041d0 2024-05-26T11:53:50.856 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:53:50.856 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":1,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","modified":"2024-05-26T11:52:04.693642Z","created":"2024-05-26T11:52:04.693642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 msgr2=0x7f35340409d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 0x7f35340409d0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3540006fd0 tx=0x7f3540006e40 comp rx=0 tx=0).stop 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 msgr2=0x7f354c19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f353c0056d0 tx=0x7f353c0057b0 comp rx=0 tx=0).stop 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 shutdown_connections 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f353403e510 0x7f35340409d0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 --2- 172.21.15.192:0/110336255 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f354c1041d0 0x7f354c19e690 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 >> 172.21.15.192:0/110336255 conn(0x7f354c0fd980 msgr2=0x7f354c104f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 shutdown_connections 2024-05-26T11:53:50.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:50.858+0000 7f35526fb700 1 -- 172.21.15.192:0/110336255 wait complete. 2024-05-26T11:53:50.860 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 1 2024-05-26T11:53:51.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:51 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/110336255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:53:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(???) e0 preinit fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:52.043 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).mds e1 new map 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: e1 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: legacy client fscid: -1 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout: No filesystems configured 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-26T11:53:52.044 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-26T11:53:52.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-26T11:53:52.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Deploying cephadm binary to smithi192 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.324 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Added host smithi192 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.325 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:53:52.326 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:52.327 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Saving service mon spec with placement smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c;count:3 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: Deploying daemon mon.c on smithi192 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/110336255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 ceph-mon[36545]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 bash[36525]: a527f7e70c411534f73c19f0ce7a15a4c382e0ebb416f69b4ceea82f449047e5 2024-05-26T11:53:52.328 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:52 smithi192 systemd[1]: Started Ceph mon.c for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:53:52.542 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-26T11:53:52.543 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-26T11:53:52.778 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:53:57.179 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1651674322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 msgr2=0x7f2e10005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1651674322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e10005a50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2e24101e70 tx=0x7f2e1400feb0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1651674322 shutdown_connections 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1651674322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e10005a50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1651674322 >> 172.21.15.192:0/1651674322 conn(0x7f2e240fda70 msgr2=0x7f2e240ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1651674322 shutdown_connections 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.177+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1651674322 wait complete. 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 Processor -- start 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 -- start start 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e2419d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2e24109e90 con 0x7f2e100035c0 2024-05-26T11:53:57.180 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.178+0000 7f2e2be7e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2e24109d10 con 0x7f2e2419dba0 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:56826/0 (socket says 172.21.15.192:56826) 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 -- 172.21.15.192:0/1412182132 learned_addr learned my addr 172.21.15.192:0/1412182132 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29c1a700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e2419d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 -- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 msgr2=0x7f2e241977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 -- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 msgr2=0x7f2e241977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29419700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29c1a700 1 -- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 msgr2=0x7f2e241977e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29c1a700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29c1a700 1 -- 172.21.15.192:0/1412182132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e2419e1e0 con 0x7f2e100035c0 2024-05-26T11:53:57.181 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e29c1a700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e2419d660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e14000c00 tx=0x7f2e140050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.179+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2e14004aa0 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.180+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e24197d80 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.180+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2e14004030 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.180+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f2e14011bb0 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.180+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e241982e0 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.180+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f2e14023070 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.181+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e24103680 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.181+0000 7f2e1affd700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 0x7f2e08040770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.181+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2e14020050 con 0x7f2e100035c0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.181+0000 7f2e29419700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 0x7f2e08040770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:53:57.183 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.181+0000 7f2e29419700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 0x7f2e08040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e20006b20 tx=0x7f2e20006ab0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:53:57.186 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.186+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2e14028020 con 0x7f2e100035c0 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: mon.a calling monitor election 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: mon.c calling monitor election 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.472 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: monmap epoch 2 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: last_changed 2024-05-26T11:53:52.049702+0000 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: min_mon_release 19 (squid) 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: election_strategy: 1 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: fsmap 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: mgrmap e13: x(active, since 50s) 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: overall HEALTH_OK 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.473 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.474 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:53:57.474 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:57.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.476+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f2e241985f0 con 0x7f2e100035c0 2024-05-26T11:53:57.476 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: mon.a calling monitor election 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: mon.c calling monitor election 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: monmap epoch 2 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: last_changed 2024-05-26T11:53:52.049702+0000 2024-05-26T11:53:57.477 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: min_mon_release 19 (squid) 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: election_strategy: 1 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: fsmap 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: mgrmap e13: x(active, since 50s) 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: overall HEALTH_OK 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:53:57.478 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:53:57.479 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.476+0000 7f2e1affd700 1 -- 172.21.15.192:0/1412182132 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f2e140103d0 con 0x7f2e100035c0 2024-05-26T11:53:57.479 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:53:57.479 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","modified":"2024-05-26T11:53:52.049702Z","created":"2024-05-26T11:52:04.693642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 msgr2=0x7f2e08040770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 0x7f2e08040770 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e20006b20 tx=0x7f2e20006ab0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 msgr2=0x7f2e2419d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e2419d660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f2e14000c00 tx=0x7f2e140050d0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 shutdown_connections 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.478+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2e0803e2b0 0x7f2e08040770 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.479+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2e100035c0 0x7f2e2419d660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.479+0000 7f2e2be7e700 1 --2- 172.21.15.192:0/1412182132 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2e2419dba0 0x7f2e241977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.479+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 >> 172.21.15.192:0/1412182132 conn(0x7f2e240fda70 msgr2=0x7f2e240fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.479+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 shutdown_connections 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:53:57.479+0000 7f2e2be7e700 1 -- 172.21.15.192:0/1412182132 wait complete. 2024-05-26T11:53:57.480 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-05-26T11:53:58.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:58 smithi192 ceph-mon[36545]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:58.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:58 smithi192 ceph-mon[36545]: Deploying daemon mon.b on smithi186 2024-05-26T11:53:58.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:58 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/1412182132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:53:58.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:53:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:58.406 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:58 smithi156 ceph-mon[32025]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:53:58.406 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:58 smithi156 ceph-mon[32025]: Deploying daemon mon.b on smithi186 2024-05-26T11:53:58.406 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:58 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/1412182132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:53:58.406 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:53:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:53:59.066 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-26T11:53:59.067 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-26T11:53:59.310 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:54:00.372 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:00 smithi192 ceph-mon[36545]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:00.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:00 smithi156 ceph-mon[32025]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1996763237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 msgr2=0x7f7c20102570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1996763237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20102570 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7c14009910 tx=0x7f7c14009c20 comp rx=0 tx=0).stop 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1996763237 shutdown_connections 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1996763237 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20102570 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1996763237 >> 172.21.15.192:0/1996763237 conn(0x7f7c200fda20 msgr2=0x7f7c200ffe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:00.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1996763237 shutdown_connections 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.706+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1996763237 wait complete. 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 Processor -- start 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 -- start start 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2010c410 con 0x7f7c20102190 2024-05-26T11:54:00.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.707+0000 7f7c24b0b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7c2010c290 con 0x7f7c20197dd0 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.192:47292/0 (socket says 172.21.15.192:47292) 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 -- 172.21.15.192:0/1565352260 learned_addr learned my addr 172.21.15.192:0/1565352260 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:56832/0 (socket says 172.21.15.192:56832) 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 -- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 msgr2=0x7f7c20198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-26T11:54:00.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 -- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 msgr2=0x7f7c20198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1dd9b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 -- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 msgr2=0x7f7c20198250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 -- 172.21.15.192:0/1565352260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c2006bcc0 con 0x7f7c20102190 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c1e59c700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c14006010 tx=0x7f7c14005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:00.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7c14005dc0 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c201a4f00 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c1400be10 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f7c14022e40 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.708+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c201a53b0 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.709+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f7c1402a070 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.709+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c20103670 con 0x7f7c20102190 2024-05-26T11:54:00.715 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.709+0000 7f7c0f7fe700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 0x7f7c08040770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:00.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.709+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f7c1401e030 con 0x7f7c20102190 2024-05-26T11:54:00.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.710+0000 7f7c1dd9b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 0x7f7c08040770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:00.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.710+0000 7f7c1dd9b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 0x7f7c08040770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009850 tx=0x7f7c10008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:00.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.715+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7c14013070 con 0x7f7c20102190 2024-05-26T11:54:00.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.997+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f7c201a5730 con 0x7f7c20102190 2024-05-26T11:54:00.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:00.998+0000 7f7c0f7fe700 1 -- 172.21.15.192:0/1565352260 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1019 (secure 0 0 0) 0x7f7c1402bd20 con 0x7f7c20102190 2024-05-26T11:54:00.999 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:54:00.999 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":2,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","modified":"2024-05-26T11:53:52.049702Z","created":"2024-05-26T11:52:04.693642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 msgr2=0x7f7c08040770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 0x7f7c08040770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7c10009850 tx=0x7f7c10008040 comp rx=0 tx=0).stop 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 msgr2=0x7f7c20197890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7c14006010 tx=0x7f7c14005030 comp rx=0 tx=0).stop 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 shutdown_connections 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c0803e2b0 0x7f7c08040770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:01.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c20102190 0x7f7c20197890 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:01.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 --2- 172.21.15.192:0/1565352260 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c20197dd0 0x7f7c20198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:01.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.000+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 >> 172.21.15.192:0/1565352260 conn(0x7f7c200fda20 msgr2=0x7f7c20103b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:01.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.001+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 shutdown_connections 2024-05-26T11:54:01.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:01.001+0000 7f7c24b0b700 1 -- 172.21.15.192:0/1565352260 wait complete. 2024-05-26T11:54:01.002 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 2 2024-05-26T11:54:01.162 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(???) e0 preinit fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).mds e1 new map 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).mds e1 print_map 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: e1 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: legacy client fscid: -1 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout: No filesystems configured 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-26T11:54:01.163 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-26T11:54:01.164 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-26T11:54:01.164 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-26T11:54:01.164 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-26T11:54:01.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:01 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/1565352260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-26T11:54:01.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi192", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Deploying cephadm binary to smithi192 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi186", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:01.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi186:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Added host smithi192 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/878272529' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi192", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi192:/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.client.admin.keyring 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:01.480 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Saving service mon spec with placement smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c;count:3 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Deploying daemon mon.c on smithi192 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/110336255' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.a calling monitor election 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.481 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.c calling monitor election 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: monmap epoch 2 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: last_changed 2024-05-26T11:53:52.049702+0000 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: min_mon_release 19 (squid) 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: election_strategy: 1 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:54:01.482 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: fsmap 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mgrmap e13: x(active, since 50s) 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: overall HEALTH_OK 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: Deploying daemon mon.b on smithi186 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/1412182132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/1565352260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 ceph-mon[36412]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-26T11:54:01.483 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 bash[36393]: 3991ed6389a20d4c9e6ba300b393fdca87826d3b82f45e2c55c2c62ae982d62b 2024-05-26T11:54:01.484 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:01 smithi186 systemd[1]: Started Ceph mon.b for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:54:01.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:01 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/1565352260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:02.527 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-26T11:54:02.528 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph mon dump -f json 2024-05-26T11:54:02.769 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:54:06.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:54:06.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: mon.a calling monitor election 2024-05-26T11:54:06.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: mon.c calling monitor election 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: mon.b calling monitor election 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: monmap epoch 3 2024-05-26T11:54:06.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: last_changed 2024-05-26T11:54:01.168038+0000 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: min_mon_release 19 (squid) 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: election_strategy: 1 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: 2: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: fsmap 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: mgrmap e13: x(active, since 59s) 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: overall HEALTH_OK 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: mon.a calling monitor election 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: mon.c calling monitor election 2024-05-26T11:54:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: mon.b calling monitor election 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: monmap epoch 3 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: last_changed 2024-05-26T11:54:01.168038+0000 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: min_mon_release 19 (squid) 2024-05-26T11:54:06.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: election_strategy: 1 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: 2: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: fsmap 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: mgrmap e13: x(active, since 59s) 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: overall HEALTH_OK 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: mon.a calling monitor election 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: mon.c calling monitor election 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: mon.b calling monitor election 2024-05-26T11:54:06.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: monmap epoch 3 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: last_changed 2024-05-26T11:54:01.168038+0000 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: created 2024-05-26T11:52:04.693642+0000 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: min_mon_release 19 (squid) 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: election_strategy: 1 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: 0: [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] mon.a 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: 1: [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] mon.c 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: 2: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: fsmap 2024-05-26T11:54:06.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: osdmap e4: 0 total, 0 up, 0 in 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: mgrmap e13: x(active, since 59s) 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: overall HEALTH_OK 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:06.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:07.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 -- 172.21.15.192:0/3980848072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 msgr2=0x7f4b10005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:07.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/3980848072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b10005610 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4b18005fa0 tx=0x7f4b1800ffc0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 -- 172.21.15.192:0/3980848072 shutdown_connections 2024-05-26T11:54:07.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/3980848072 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b10005610 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.172 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 -- 172.21.15.192:0/3980848072 >> 172.21.15.192:0/3980848072 conn(0x7f4b280fda40 msgr2=0x7f4b280ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:07.173 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 -- 172.21.15.192:0/3980848072 shutdown_connections 2024-05-26T11:54:07.173 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.171+0000 7f4b2caa8700 1 -- 172.21.15.192:0/3980848072 wait complete. 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.172+0000 7f4b2caa8700 1 Processor -- start 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.172+0000 7f4b2caa8700 1 -- start start 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b2819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 0x7f4b28197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28109e40 con 0x7f4b281021b0 2024-05-26T11:54:07.175 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28109cc0 con 0x7f4b2819dc10 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2caa8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4b28109fc0 con 0x7f4b28197d90 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b26d9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b26d9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:59718/0 (socket says 172.21.15.192:59718) 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 0x7f4b28197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b26d9d700 1 -- 172.21.15.192:0/1163562880 learned_addr learned my addr 172.21.15.192:0/1163562880 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b2659c700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b2819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 -- 172.21.15.192:0/1163562880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 msgr2=0x7f4b28198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 -- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 msgr2=0x7f4b2819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b2819d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 -- 172.21.15.192:0/1163562880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b281a4530 con 0x7f4b2819dc10 2024-05-26T11:54:07.176 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b26d9d700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.173+0000 7f4b25d9b700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 0x7f4b28197850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00b550 tx=0x7f4b1c00b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.174+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c004230 con 0x7f4b2819dc10 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.174+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4b1c0043d0 con 0x7f4b2819dc10 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.174+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b1c0119e0 con 0x7f4b2819dc10 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.174+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b281a47c0 con 0x7f4b2819dc10 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.174+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b281a4c80 con 0x7f4b2819dc10 2024-05-26T11:54:07.177 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.175+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b28103690 con 0x7f4b2819dc10 2024-05-26T11:54:07.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.175+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7f4b1c011ba0 con 0x7f4b2819dc10 2024-05-26T11:54:07.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.175+0000 7f4b177fe700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 0x7f4b08040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:07.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.175+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f4b1c052300 con 0x7f4b2819dc10 2024-05-26T11:54:07.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.175+0000 7f4b2659c700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 0x7f4b08040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:07.178 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.176+0000 7f4b2659c700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 0x7f4b08040e60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800fb20 tx=0x7f4b1801d040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:07.185 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.185+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4b1c018350 con 0x7f4b2819dc10 2024-05-26T11:54:07.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:07.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:07.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:07.490 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.490+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f4b281a5060 con 0x7f4b2819dc10 2024-05-26T11:54:07.490 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:07.491 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:07.491 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:07.491 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.490+0000 7f4b177fe700 1 -- 172.21.15.192:0/1163562880 <== mon.2 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1289 (secure 0 0 0) 0x7f4b1c021d20 con 0x7f4b2819dc10 2024-05-26T11:54:07.491 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-05-26T11:54:07.491 INFO:teuthology.orchestra.run.smithi192.stdout:{"epoch":3,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","modified":"2024-05-26T11:54:01.168038Z","created":"2024-05-26T11:52:04.693642Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:3300","nonce":0},{"type":"v1","addr":"172.21.15.156:6789","nonce":0}]},"addr":"172.21.15.156:6789/0","public_addr":"172.21.15.156:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:3300","nonce":0},{"type":"v1","addr":"172.21.15.192:6789","nonce":0}]},"addr":"172.21.15.192:6789/0","public_addr":"172.21.15.192:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:3300","nonce":0},{"type":"v1","addr":"172.21.15.186:6789","nonce":0}]},"addr":"172.21.15.186:6789/0","public_addr":"172.21.15.186:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-26T11:54:07.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.492+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 msgr2=0x7f4b08040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:07.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.492+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 0x7f4b08040e60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4b1800fb20 tx=0x7f4b1801d040 comp rx=0 tx=0).stop 2024-05-26T11:54:07.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.492+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 msgr2=0x7f4b28197850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:07.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 0x7f4b28197850 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f4b1c00b550 tx=0x7f4b1c00b860 comp rx=0 tx=0).stop 2024-05-26T11:54:07.493 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 shutdown_connections 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4b0803e9a0 0x7f4b08040e60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4b28197d90 0x7f4b28198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4b281021b0 0x7f4b2819d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 --2- 172.21.15.192:0/1163562880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4b2819dc10 0x7f4b28197850 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 >> 172.21.15.192:0/1163562880 conn(0x7f4b280fda40 msgr2=0x7f4b280fedf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 shutdown_connections 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:07.493+0000 7f4b2caa8700 1 -- 172.21.15.192:0/1163562880 wait complete. 2024-05-26T11:54:07.494 INFO:teuthology.orchestra.run.smithi192.stderr:dumped monmap epoch 3 2024-05-26T11:54:07.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:07.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:07.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:54:08.312 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-26T11:54:08.312 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph config generate-minimal-conf 2024-05-26T11:54:08.570 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/1163562880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:08.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/1163562880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.681 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:08.682 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/1163562880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:08.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:09.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi156:/etc/ceph/ceph.conf 2024-05-26T11:54:09.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Reconfiguring mon.a (unknown last config time)... 2024-05-26T11:54:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:09 smithi186 ceph-mon[36412]: Reconfiguring daemon mon.a on smithi156 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi156:/etc/ceph/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Reconfiguring mon.a (unknown last config time)... 2024-05-26T11:54:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:09 smithi156 ceph-mon[32025]: Reconfiguring daemon mon.a on smithi156 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi156:/etc/ceph/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi186:/etc/ceph/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi192:/etc/ceph/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Updating smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/config/ceph.conf 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Reconfiguring mon.a (unknown last config time)... 2024-05-26T11:54:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:09 smithi192 ceph-mon[36545]: Reconfiguring daemon mon.a on smithi156 2024-05-26T11:54:10.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 -- 172.21.15.156:0/2807676798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f41020b0 msgr2=0x7ff5f4102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:10.345 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 --2- 172.21.15.156:0/2807676798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f41020b0 0x7ff5f4102490 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4009910 tx=0x7ff5e4009c20 comp rx=0 tx=0).stop 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 -- 172.21.15.156:0/2807676798 shutdown_connections 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 --2- 172.21.15.156:0/2807676798 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f41020b0 0x7ff5f4102490 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 -- 172.21.15.156:0/2807676798 >> 172.21.15.156:0/2807676798 conn(0x7ff5f40fd980 msgr2=0x7ff5f40ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 -- 172.21.15.156:0/2807676798 shutdown_connections 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.344+0000 7ff5f892c700 1 -- 172.21.15.156:0/2807676798 wait complete. 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 Processor -- start 2024-05-26T11:54:10.346 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 -- start start 2024-05-26T11:54:10.347 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f41020b0 0x7ff5f419e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 0x7ff5f4198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f410c330 con 0x7ff5f4198f90 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f410c1b0 con 0x7ff5f41020b0 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f892c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f410c4b0 con 0x7ff5f419edd0 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f2d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59306/0 (socket says 172.21.15.156:59306) 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.345+0000 7ff5f2d9d700 1 -- 172.21.15.156:0/559743084 learned_addr learned my addr 172.21.15.156:0/559743084 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f259c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f41020b0 0x7ff5f419e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f1d9b700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 0x7ff5f4198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 -- 172.21.15.156:0/559743084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 msgr2=0x7ff5f4198a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:10.348 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 0x7ff5f4198a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 -- 172.21.15.156:0/559743084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f41020b0 msgr2=0x7ff5f419e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f41020b0 0x7ff5f419e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 -- 172.21.15.156:0/559743084 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f41a5760 con 0x7ff5f4198f90 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f1d9b700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 0x7ff5f4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f2d9d700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00ba50 tx=0x7ff5ec00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec019070 con 0x7ff5f4198f90 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5ec014440 con 0x7ff5f4198f90 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f41a5990 con 0x7ff5f4198f90 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec007590 con 0x7ff5f4198f90 2024-05-26T11:54:10.349 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.346+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f41a5ed0 con 0x7ff5f4198f90 2024-05-26T11:54:10.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.347+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7ff5ec01c070 con 0x7ff5f4198f90 2024-05-26T11:54:10.350 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.348+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f4064ba0 con 0x7ff5f4198f90 2024-05-26T11:54:10.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.348+0000 7ff5e37fe700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 0x7ff5dc040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:10.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.348+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff5ec051eb0 con 0x7ff5f4198f90 2024-05-26T11:54:10.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.348+0000 7ff5f259c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 0x7ff5dc040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:10.351 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.348+0000 7ff5f259c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 0x7ff5dc040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4006010 tx=0x7ff5e401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:10.353 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.353+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff5ec022170 con 0x7ff5f4198f90 2024-05-26T11:54:10.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.569+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7ff5f41a6250 con 0x7ff5f4198f90 2024-05-26T11:54:10.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.570+0000 7ff5e37fe700 1 -- 172.21.15.156:0/559743084 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+277 (secure 0 0 0) 0x7ff5ec012c40 con 0x7ff5f4198f90 2024-05-26T11:54:10.571 INFO:teuthology.orchestra.run.smithi156.stdout:# minimal ceph.conf for 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:10.571 INFO:teuthology.orchestra.run.smithi156.stdout:[global] 2024-05-26T11:54:10.571 INFO:teuthology.orchestra.run.smithi156.stdout: fsid = 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T11:54:10.571 INFO:teuthology.orchestra.run.smithi156.stdout: mon_host = [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 msgr2=0x7ff5dc040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 0x7ff5dc040e60 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff5e4006010 tx=0x7ff5e401a040 comp rx=0 tx=0).stop 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 msgr2=0x7ff5f4199440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00ba50 tx=0x7ff5ec00be10 comp rx=0 tx=0).stop 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 shutdown_connections 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5dc03e9a0 0x7ff5dc040e60 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5f4198f90 0x7ff5f4199440 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5f41020b0 0x7ff5f419e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.573 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 --2- 172.21.15.156:0/559743084 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5f419edd0 0x7ff5f4198a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:10.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 >> 172.21.15.156:0/559743084 conn(0x7ff5f40fd980 msgr2=0x7ff5f406b200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:10.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 shutdown_connections 2024-05-26T11:54:10.574 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:10.572+0000 7ff5f892c700 1 -- 172.21.15.156:0/559743084 wait complete. 2024-05-26T11:54:10.712 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:10 smithi156 ceph-mon[32025]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:10.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:10 smithi186 ceph-mon[36412]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:10.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:10 smithi192 ceph-mon[36545]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/559743084' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:11.591 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.617 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-26T11:54:11.617 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:54:11.618 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.conf 2024-05-26T11:54:11.645 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:54:11.645 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:54:11.710 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:54:11.710 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.conf 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/559743084' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.736 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:54:11.736 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:54:11.801 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:54:11.801 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.conf 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/559743084' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:11.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:11.832 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:54:11.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-26T11:54:11.899 INFO:tasks.cephadm:Adding mgr.x on smithi156 2024-05-26T11:54:11.899 INFO:tasks.cephadm:Adding mgr.y on smithi186 2024-05-26T11:54:11.899 INFO:tasks.cephadm:Adding mgr.z on smithi192 2024-05-26T11:54:11.899 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch apply mgr '3;smithi156=x;smithi186=y;smithi192=z' 2024-05-26T11:54:12.174 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:54:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:12 smithi192 ceph-mon[36545]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:12 smithi192 ceph-mon[36545]: Reconfiguring mon.b (monmap changed)... 2024-05-26T11:54:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:12 smithi192 ceph-mon[36545]: Reconfiguring daemon mon.b on smithi186 2024-05-26T11:54:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:12 smithi186 ceph-mon[36412]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:12 smithi186 ceph-mon[36412]: Reconfiguring mon.b (monmap changed)... 2024-05-26T11:54:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:12 smithi186 ceph-mon[36412]: Reconfiguring daemon mon.b on smithi186 2024-05-26T11:54:12.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:12 smithi156 ceph-mon[32025]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:12.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:12 smithi156 ceph-mon[32025]: Reconfiguring mon.b (monmap changed)... 2024-05-26T11:54:12.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:12 smithi156 ceph-mon[32025]: Reconfiguring daemon mon.b on smithi186 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 -- 172.21.15.192:0/3018186103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc94103b30 msgr2=0x7fbc94105f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/3018186103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc94103b30 0x7fbc94105f20 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbc84009910 tx=0x7fbc84009c20 comp rx=0 tx=0).stop 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 -- 172.21.15.192:0/3018186103 shutdown_connections 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/3018186103 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc94103b30 0x7fbc94105f20 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 -- 172.21.15.192:0/3018186103 >> 172.21.15.192:0/3018186103 conn(0x7fbc940fd540 msgr2=0x7fbc940ff980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:13.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 -- 172.21.15.192:0/3018186103 shutdown_connections 2024-05-26T11:54:13.702 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.700+0000 7fbc9a20f700 1 -- 172.21.15.192:0/3018186103 wait complete. 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 Processor -- start 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 -- start start 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc94103b30 0x7fbc94103490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc94101ae0 0x7fbc94101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94108f80 con 0x7fbc941024a0 2024-05-26T11:54:13.703 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94108e00 con 0x7fbc94101ae0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.701+0000 7fbc9a20f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbc94109100 con 0x7fbc94103b30 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc937fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc94103b30 0x7fbc94103490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.192:47396/0 (socket says 172.21.15.192:47396) 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 -- 172.21.15.192:0/1100906399 learned_addr learned my addr 172.21.15.192:0/1100906399 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc92ffd700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc94101ae0 0x7fbc94101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 -- 172.21.15.192:0/1100906399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc94103b30 msgr2=0x7fbc94103490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc94103b30 0x7fbc94103490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 -- 172.21.15.192:0/1100906399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc94101ae0 msgr2=0x7fbc94101f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc94101ae0 0x7fbc94101f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 -- 172.21.15.192:0/1100906399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc941a2db0 con 0x7fbc941024a0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc93fff700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00ba50 tx=0x7fbc8c00bd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c019070 con 0x7fbc941024a0 2024-05-26T11:54:13.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc941a30a0 con 0x7fbc941024a0 2024-05-26T11:54:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbc8c014440 con 0x7fbc941024a0 2024-05-26T11:54:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc8c00ce60 con 0x7fbc941024a0 2024-05-26T11:54:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.702+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc941a3660 con 0x7fbc941024a0 2024-05-26T11:54:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.703+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fbc8c01c070 con 0x7fbc941024a0 2024-05-26T11:54:13.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.704+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc94064ba0 con 0x7fbc941024a0 2024-05-26T11:54:13.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.704+0000 7fbc90ff9700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 0x7fbc7c040e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:13.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.704+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fbc8c051dc0 con 0x7fbc941024a0 2024-05-26T11:54:13.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.704+0000 7fbc937fe700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 0x7fbc7c040e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:13.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.704+0000 7fbc937fe700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 0x7fbc7c040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc84005190 tx=0x7fbc8401a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:13.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.709+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbc8c00f160 con 0x7fbc941024a0 2024-05-26T11:54:13.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: Reconfiguring mon.c (monmap changed)... 2024-05-26T11:54:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:13.785 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:13 smithi192 ceph-mon[36545]: Reconfiguring daemon mon.c on smithi192 2024-05-26T11:54:13.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: Reconfiguring mon.c (monmap changed)... 2024-05-26T11:54:13.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:13.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:13.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:13.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:13 smithi156 ceph-mon[32025]: Reconfiguring daemon mon.c on smithi192 2024-05-26T11:54:13.928 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.927+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi156=x;smithi186=y;smithi192=z", "target": ["mon-mgr", ""]}) -- 0x7fbc94063a10 con 0x7fbc7c03e9a0 2024-05-26T11:54:13.935 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.934+0000 7fbc90ff9700 1 -- 172.21.15.192:0/1100906399 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fbc94063a10 con 0x7fbc7c03e9a0 2024-05-26T11:54:13.935 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mgr update... 2024-05-26T11:54:13.937 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 msgr2=0x7fbc7c040e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:13.937 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 0x7fbc7c040e60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbc84005190 tx=0x7fbc8401a040 comp rx=0 tx=0).stop 2024-05-26T11:54:13.937 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 msgr2=0x7fbc941a27f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c00ba50 tx=0x7fbc8c00bd60 comp rx=0 tx=0).stop 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 shutdown_connections 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbc7c03e9a0 0x7fbc7c040e60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbc941024a0 0x7fbc941a27f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbc94103b30 0x7fbc94103490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 --2- 172.21.15.192:0/1100906399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc94101ae0 0x7fbc94101f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 >> 172.21.15.192:0/1100906399 conn(0x7fbc940fd540 msgr2=0x7fbc940fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 shutdown_connections 2024-05-26T11:54:13.938 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:54:13.937+0000 7fbc9a20f700 1 -- 172.21.15.192:0/1100906399 wait complete. 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: Reconfiguring mon.c (monmap changed)... 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-26T11:54:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:13.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:13 smithi186 ceph-mon[36412]: Reconfiguring daemon mon.c on smithi192 2024-05-26T11:54:14.701 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:14 smithi192 ceph-mon[36545]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:14.701 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:14 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:14.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:14 smithi156 ceph-mon[32025]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:14.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:14 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:14.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:14 smithi186 ceph-mon[36412]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:14.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:14 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.386 DEBUG:teuthology.orchestra.run.smithi186:mgr.y> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.y.service 2024-05-26T11:54:15.389 DEBUG:teuthology.orchestra.run.smithi192:mgr.z> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.z.service 2024-05-26T11:54:15.391 INFO:tasks.cephadm:Deploying OSDs... 2024-05-26T11:54:15.392 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T11:54:15.392 DEBUG:teuthology.orchestra.run.smithi156:> dd if=/scratch_devs of=/dev/stdout 2024-05-26T11:54:15.408 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-26T11:54:15.408 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_1 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-05-26 11:53:16.817873100 +0000 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-05-26 11:48:03.936456628 +0000 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-05-26 11:48:03.936456628 +0000 2024-05-26T11:54:15.464 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-05-26 11:48:03.936456628 +0000 2024-05-26T11:54:15.464 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-26T11:54:15.529 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-05-26T11:54:15.529 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-05-26T11:54:15.529 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.6898e-05 s, 5.3 MB/s 2024-05-26T11:54:15.531 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-26T11:54:15.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:15 smithi192 ceph-mon[36545]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi156=x;smithi186=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:15 smithi192 ceph-mon[36545]: Saving service mgr spec with placement smithi156=x;smithi186=y;smithi192=z;count:3 2024-05-26T11:54:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:15.586 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_2 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-05-26 11:53:16.848872650 +0000 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-05-26 11:48:04.246451922 +0000 2024-05-26T11:54:15.643 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-05-26 11:48:04.246451922 +0000 2024-05-26T11:54:15.644 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-05-26 11:48:04.246451922 +0000 2024-05-26T11:54:15.644 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-26T11:54:15.708 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-05-26T11:54:15.708 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-05-26T11:54:15.708 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 8.5874e-05 s, 6.0 MB/s 2024-05-26T11:54:15.709 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-26T11:54:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:15 smithi186 ceph-mon[36412]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi156=x;smithi186=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:15 smithi186 ceph-mon[36412]: Saving service mgr spec with placement smithi156=x;smithi186=y;smithi192=z;count:3 2024-05-26T11:54:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:15.765 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_3 2024-05-26T11:54:15.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:15 smithi156 ceph-mon[32025]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi156=x;smithi186=y;smithi192=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:15.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:15 smithi156 ceph-mon[32025]: Saving service mgr spec with placement smithi156=x;smithi186=y;smithi192=z;count:3 2024-05-26T11:54:15.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:15.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-05-26 11:53:16.877872229 +0000 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-05-26 11:48:04.558447186 +0000 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-05-26 11:48:04.558447186 +0000 2024-05-26T11:54:15.810 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-05-26 11:48:04.558447186 +0000 2024-05-26T11:54:15.811 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-26T11:54:15.873 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-05-26T11:54:15.873 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-05-26T11:54:15.873 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 0.000102702 s, 5.0 MB/s 2024-05-26T11:54:15.874 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-26T11:54:15.930 DEBUG:teuthology.orchestra.run.smithi156:> stat /dev/vg_nvme/lv_4 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Access: 2024-05-26 11:53:16.909871765 +0000 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Modify: 2024-05-26 11:48:04.872442420 +0000 2024-05-26T11:54:15.986 INFO:teuthology.orchestra.run.smithi156.stdout:Change: 2024-05-26 11:48:04.872442420 +0000 2024-05-26T11:54:15.987 INFO:teuthology.orchestra.run.smithi156.stdout: Birth: 2024-05-26 11:48:04.872442420 +0000 2024-05-26T11:54:15.987 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-26T11:54:16.049 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records in 2024-05-26T11:54:16.049 INFO:teuthology.orchestra.run.smithi156.stderr:1+0 records out 2024-05-26T11:54:16.049 INFO:teuthology.orchestra.run.smithi156.stderr:512 bytes copied, 9.0707e-05 s, 5.6 MB/s 2024-05-26T11:54:16.051 DEBUG:teuthology.orchestra.run.smithi156:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-26T11:54:16.107 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2024-05-26T11:54:16.108 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2024-05-26T11:54:16.127 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-26T11:54:16.127 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2024-05-26T11:54:16.185 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-26T11:54:16.185 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:16.185 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-26T11:54:16.185 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:16.186 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:16.186 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-26 11:54:00.609100304 +0000 2024-05-26T11:54:16.186 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-26 11:48:19.539623939 +0000 2024-05-26T11:54:16.186 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-26 11:48:19.539623939 +0000 2024-05-26T11:54:16.186 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-26 11:48:19.539623939 +0000 2024-05-26T11:54:16.186 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-26T11:54:16.251 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-26T11:54:16.252 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-26T11:54:16.252 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000356104 s, 1.4 MB/s 2024-05-26T11:54:16.253 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-26T11:54:16.311 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-26 11:54:00.610100287 +0000 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-26 11:48:19.871618090 +0000 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-26 11:48:19.871618090 +0000 2024-05-26T11:54:16.368 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-26 11:48:19.871618090 +0000 2024-05-26T11:54:16.369 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-26T11:54:16.434 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-26T11:54:16.434 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-26T11:54:16.434 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000222746 s, 2.3 MB/s 2024-05-26T11:54:16.435 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-26T11:54:16.493 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-26 11:54:00.610100287 +0000 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-26 11:48:20.191612452 +0000 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-26 11:48:20.191612452 +0000 2024-05-26T11:54:16.550 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-26 11:48:20.191612452 +0000 2024-05-26T11:54:16.551 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:16.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:16.612 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:16.613 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.614 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-26T11:54:16.615 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-26T11:54:16.615 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000205798 s, 2.5 MB/s 2024-05-26T11:54:16.617 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-26T11:54:16.673 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2024-05-26T11:54:16.729 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2024-05-26 11:54:00.610100287 +0000 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2024-05-26 11:48:20.507606885 +0000 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2024-05-26 11:48:20.507606885 +0000 2024-05-26T11:54:16.730 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2024-05-26 11:48:20.507606885 +0000 2024-05-26T11:54:16.730 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-26T11:54:16.792 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2024-05-26T11:54:16.793 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2024-05-26T11:54:16.793 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000195046 s, 2.6 MB/s 2024-05-26T11:54:16.794 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-26T11:54:16.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:16.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:16.849 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-05-26T11:54:16.850 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-05-26T11:54:16.866 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-26T11:54:16.866 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-05-26 11:53:50.869745576 +0000 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-05-26 11:48:25.017767845 +0000 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-05-26 11:48:25.017767845 +0000 2024-05-26T11:54:16.921 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-05-26 11:48:25.017767845 +0000 2024-05-26T11:54:16.922 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-26T11:54:16.985 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-05-26T11:54:16.985 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-05-26T11:54:16.985 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000246543 s, 2.1 MB/s 2024-05-26T11:54:16.986 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-26T11:54:17.042 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-05-26T11:54:17.096 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-05-26 11:53:50.869745576 +0000 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-05-26 11:48:25.375760385 +0000 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-05-26 11:48:25.375760385 +0000 2024-05-26T11:54:17.097 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-05-26 11:48:25.375760385 +0000 2024-05-26T11:54:17.097 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-26T11:54:17.160 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-05-26T11:54:17.160 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-05-26T11:54:17.160 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000192371 s, 2.7 MB/s 2024-05-26T11:54:17.161 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-26T11:54:17.217 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-05-26T11:54:17.272 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-26T11:54:17.272 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:17.272 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-05-26T11:54:17.272 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:17.273 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:17.273 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-05-26 11:53:50.869745576 +0000 2024-05-26T11:54:17.273 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-05-26 11:48:25.720753196 +0000 2024-05-26T11:54:17.273 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-05-26 11:48:25.720753196 +0000 2024-05-26T11:54:17.273 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-05-26 11:48:25.720753196 +0000 2024-05-26T11:54:17.273 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-26T11:54:17.334 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-05-26T11:54:17.335 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-05-26T11:54:17.335 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000172738 s, 3.0 MB/s 2024-05-26T11:54:17.336 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-26T11:54:17.392 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-05-26T11:54:17.447 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-05-26 11:53:50.870745556 +0000 2024-05-26T11:54:17.448 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-05-26 11:48:26.045746424 +0000 2024-05-26T11:54:17.448 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-05-26 11:48:26.045746424 +0000 2024-05-26T11:54:17.448 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-05-26 11:48:26.045746424 +0000 2024-05-26T11:54:17.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-26T11:54:17.558 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-05-26T11:54:17.559 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-05-26T11:54:17.559 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.00021815 s, 2.3 MB/s 2024-05-26T11:54:17.560 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-26T11:54:17.576 INFO:tasks.cephadm:Deploying osd.0 on smithi156 with /dev/vg_nvme/lv_4... 2024-05-26T11:54:17.576 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-26T11:54:17.762 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:17 smithi192 ceph-mon[36545]: Deploying daemon mgr.z on smithi192 2024-05-26T11:54:17.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:17 smithi156 ceph-mon[32025]: Deploying daemon mgr.z on smithi192 2024-05-26T11:54:17.806 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:17 smithi186 ceph-mon[36412]: Deploying daemon mgr.z on smithi192 2024-05-26T11:54:17.823 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:54:18.317 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 systemd[1]: Started Ceph mgr.z for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:54:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:18.317+0000 7efeb4be9200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:18.502+0000 7efeb4be9200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:18.911 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:18.605+0000 7efeb4be9200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T11:54:18.912 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:18.911+0000 7efeb4be9200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:18.954 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:19.309 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:18.996+0000 7efeb4be9200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T11:54:19.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:19 smithi192 ceph-mon[36545]: Deploying daemon mgr.y on smithi186 2024-05-26T11:54:19.660 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-26T11:54:19.660 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-26T11:54:19.660 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-05-26T11:54:19.661 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-05-26T11:54:19.661 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00859571 s, 1.2 GB/s 2024-05-26T11:54:19.661 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-05-26T11:54:19.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:19 smithi186 ceph-mon[36412]: Deploying daemon mgr.y on smithi186 2024-05-26T11:54:19.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:19 smithi156 ceph-mon[32025]: Deploying daemon mgr.y on smithi186 2024-05-26T11:54:20.272 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_4 2024-05-26T11:54:20.476 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 systemd[1]: Starting Ceph mgr.y for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:54:20.508 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:20.185+0000 7efeb4be9200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T11:54:20.508 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:20.352+0000 7efeb4be9200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T11:54:20.508 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:54:20.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:20 smithi156 ceph-mon[32025]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:20.739 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:20 smithi186 ceph-mon[36412]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 podman[37779]: 2024-05-26 11:54:20.487622349 +0000 UTC m=+0.105009644 container create 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 podman[37779]: 2024-05-26 11:54:20.399829483 +0000 UTC m=+0.017216756 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 podman[37779]: 2024-05-26 11:54:20.673156564 +0000 UTC m=+0.290543826 container init 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 podman[37779]: 2024-05-26 11:54:20.676441949 +0000 UTC m=+0.293829203 container start 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 ceph-mgr[37798]: -- 172.21.15.186:0/612618665 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5645c04d65a0 con 0x5645c04aa800 2024-05-26T11:54:20.740 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 bash[37779]: 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 2024-05-26T11:54:20.808 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:20 smithi192 ceph-mon[36545]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:20.808 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:20.521+0000 7efeb4be9200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T11:54:21.019 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 systemd[1]: Started Ceph mgr.y for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:54:21.019 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:20.830+0000 7fc49a10a200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-26T11:54:21.019 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:21.018+0000 7fc49a10a200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:54:21.071 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:20.807+0000 7efeb4be9200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T11:54:21.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:20.899+0000 7efeb4be9200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T11:54:21.455 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:21.126+0000 7fc49a10a200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-26T11:54:21.455 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:21.455+0000 7fc49a10a200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-26T11:54:21.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:21.114+0000 7efeb4be9200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T11:54:21.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:21.314+0000 7efeb4be9200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:54:21.726 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:21.557+0000 7fc49a10a200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-26T11:54:22.003 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:22.003 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.004 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.004 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.004 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.004 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:22.004 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.027+0000 7fcc7633e700 1 -- 172.21.15.156:0/4034445066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc701036a0 msgr2=0x7fcc70103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.027+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4034445066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc701036a0 0x7fcc70103b00 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc580099c0 tx=0x7fcc58009cd0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- 172.21.15.156:0/4034445066 shutdown_connections 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4034445066 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc70104040 0x7fcc7010a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4034445066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701024a0 0x7fcc701028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4034445066 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc701036a0 0x7fcc70103b00 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- 172.21.15.156:0/4034445066 >> 172.21.15.156:0/4034445066 conn(0x7fcc700fda40 msgr2=0x7fcc700ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- 172.21.15.156:0/4034445066 shutdown_connections 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- 172.21.15.156:0/4034445066 wait complete. 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 Processor -- start 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- start start 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 0x7fcc701985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 0x7fcc70198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7010df20 con 0x7fcc70104040 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7010dda0 con 0x7fcc701036a0 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.028+0000 7fcc7633e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7010e0a0 con 0x7fcc701024a0 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 0x7fcc70198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:22.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc6ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 0x7fcc701985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42036/0 (socket says 172.21.15.156:42036) 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc67fff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 0x7fcc70198b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.156:40000/0 (socket says 172.21.15.156:40000) 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc6ffff700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 0x7fcc701985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.156:34012/0 (socket says 172.21.15.156:34012) 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 -- 172.21.15.156:0/4140884318 learned_addr learned my addr 172.21.15.156:0/4140884318 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 -- 172.21.15.156:0/4140884318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 msgr2=0x7fcc701985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 0x7fcc701985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 -- 172.21.15.156:0/4140884318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 msgr2=0x7fcc70198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 0x7fcc70198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:22.031 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 -- 172.21.15.156:0/4140884318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc701a0220 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc748db700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc68007f10 tx=0x7fcc6800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc68004bb0 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc701a0510 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc68004d50 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc680058e0 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.029+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc701a0a00 con 0x7fcc70104040 2024-05-26T11:54:22.032 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.030+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 13) ==== 50279+0+0 (secure 0 0 0) 0x7fcc6801a020 con 0x7fcc70104040 2024-05-26T11:54:22.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.030+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc7006a230 con 0x7fcc70104040 2024-05-26T11:54:22.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.030+0000 7fcc6dffb700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 0x7fcc5c040c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:22.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.030+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcc680521f0 con 0x7fcc70104040 2024-05-26T11:54:22.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.030+0000 7fcc6ffff700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 0x7fcc5c040c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:22.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.031+0000 7fcc6ffff700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 0x7fcc5c040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc70103500 tx=0x7fcc60007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:22.038 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.036+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcc68010070 con 0x7fcc70104040 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:21.961+0000 7efeb4be9200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:22.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:22.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:22.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:22.250+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fcc70063980 con 0x7fcc5c03e750 2024-05-26T11:54:22.571 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:22.183+0000 7efeb4be9200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:22.617+0000 7efeb4be9200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:22.713+0000 7efeb4be9200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: Reconfiguring mgr.x (unknown last config time)... 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:22 smithi192 ceph-mon[36545]: Reconfiguring daemon mgr.x on smithi156 2024-05-26T11:54:23.135 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:22.792+0000 7fc49a10a200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-26T11:54:23.135 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:22.963+0000 7fc49a10a200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:23.135+0000 7fc49a10a200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: Reconfiguring mgr.x (unknown last config time)... 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.137 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:22 smithi186 ceph-mon[36412]: Reconfiguring daemon mgr.x on smithi156 2024-05-26T11:54:23.148 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='client.14229 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:23.148 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:23.148 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:23.148 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: Reconfiguring mgr.x (unknown last config time)... 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:23.149 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:22 smithi156 ceph-mon[32025]: Reconfiguring daemon mgr.x on smithi156 2024-05-26T11:54:23.438 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.085+0000 7efeb4be9200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T11:54:23.438 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.438+0000 7efeb4be9200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T11:54:23.718 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:23.422+0000 7fc49a10a200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-26T11:54:23.718 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:23.514+0000 7fc49a10a200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-26T11:54:23.719 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:23.718+0000 7fc49a10a200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-26T11:54:23.768 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.529+0000 7efeb4be9200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T11:54:23.976 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:23.910+0000 7fc49a10a200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-26T11:54:24.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.768+0000 7efeb4be9200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T11:54:24.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.899+0000 7efeb4be9200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T11:54:24.072 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:23.994+0000 7efeb4be9200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T11:54:24.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:24 smithi156 ceph-mon[32025]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:24.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:24 smithi192 ceph-mon[36545]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:24.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:24.321+0000 7efeb4be9200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T11:54:24.572 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:24.413+0000 7efeb4be9200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T11:54:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:24 smithi186 ceph-mon[36412]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:24.726 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:24 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:24.510+0000 7fc49a10a200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-26T11:54:25.153 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:24 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:24.730+0000 7fc49a10a200 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-26T11:54:25.322 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:25.011+0000 7efeb4be9200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T11:54:25.322 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 11:54:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z[38543]: 2024-05-26T11:54:25.225+0000 7efeb4be9200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T11:54:25.476 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:25 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:25.152+0000 7fc49a10a200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-26T11:54:25.476 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:25 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:25.248+0000 7fc49a10a200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-26T11:54:25.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:25.847+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mgrmap(e 14) ==== 99938+0+0 (secure 0 0 0) 0x7fcc680133f0 con 0x7fcc70104040 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:25 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:25.618+0000 7fc49a10a200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:25.973 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: Standby manager daemon z started 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-26T11:54:25.974 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:25 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: Standby manager daemon z started 2024-05-26T11:54:26.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-26T11:54:26.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:26.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-26T11:54:26.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:25 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: Standby manager daemon z started 2024-05-26T11:54:26.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-26T11:54:26.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:26.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-26T11:54:26.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:25 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.192:0/2157329305' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:26.226 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:25 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:25.972+0000 7fc49a10a200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-26T11:54:26.226 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.064+0000 7fc49a10a200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-26T11:54:26.726 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.306+0000 7fc49a10a200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-26T11:54:26.726 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.437+0000 7fc49a10a200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-26T11:54:26.726 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.533+0000 7fc49a10a200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-26T11:54:27.226 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.862+0000 7fc49a10a200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-26T11:54:27.226 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:26.953+0000 7fc49a10a200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]: dispatch 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]': finished 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: osdmap e5: 1 total, 0 up, 1 in 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:27.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:26 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1911156894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:27.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-26T11:54:27.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-26T11:54:27.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]: dispatch 2024-05-26T11:54:27.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]': finished 2024-05-26T11:54:27.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: osdmap e5: 1 total, 0 up, 1 in 2024-05-26T11:54:27.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:27.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:26 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1911156894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: mgrmap e14: x(active, since 79s), standbys: z 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]: dispatch 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1781361228' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e405c54e-bacd-4ac7-90d1-3969a01a02f1"}]': finished 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: osdmap e5: 1 total, 0 up, 1 in 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:27.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:26 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1911156894' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:27.871 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:27 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:27.558+0000 7fc49a10a200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-26T11:54:27.871 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 11:54:27 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y[37794]: 2024-05-26T11:54:27.773+0000 7fc49a10a200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-26T11:54:27.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:27.971+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mon.0 v2:172.21.15.156:3300/0 8 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7fcc68004f00 con 0x7fcc70104040 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: Standby manager daemon y started 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-26T11:54:28.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:27 smithi186 ceph-mon[36412]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: Standby manager daemon y started 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-26T11:54:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:27 smithi156 ceph-mon[32025]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: Standby manager daemon y started 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-26T11:54:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:27 smithi192 ceph-mon[36545]: from='mgr.? 172.21.15.186:0/3088849398' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-26T11:54:29.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:28 smithi156 ceph-mon[32025]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-26T11:54:29.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-26T11:54:29.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:28 smithi192 ceph-mon[36545]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-26T11:54:29.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-26T11:54:29.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:28 smithi186 ceph-mon[36412]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-05-26T11:54:29.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-26T11:54:30.275 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:29 smithi156 ceph-mon[32025]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:29 smithi192 ceph-mon[36545]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:29 smithi186 ceph-mon[36412]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:32.266 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:31 smithi156 ceph-mon[32025]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:32.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:31 smithi192 ceph-mon[36545]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:32.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:31 smithi186 ceph-mon[36412]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:34.258 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:33 smithi156 ceph-mon[32025]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:34.258 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-26T11:54:34.259 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:34.259 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:33 smithi156 ceph-mon[32025]: Deploying daemon osd.0 on smithi156 2024-05-26T11:54:34.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:33 smithi192 ceph-mon[36545]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:34.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-26T11:54:34.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:34.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:33 smithi192 ceph-mon[36545]: Deploying daemon osd.0 on smithi156 2024-05-26T11:54:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:33 smithi186 ceph-mon[36412]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-26T11:54:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:33 smithi186 ceph-mon[36412]: Deploying daemon osd.0 on smithi156 2024-05-26T11:54:36.286 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:35 smithi156 ceph-mon[32025]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:36.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:35 smithi192 ceph-mon[36545]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:35 smithi186 ceph-mon[36412]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:38.245 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:37 smithi156 ceph-mon[32025]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:38.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:37 smithi192 ceph-mon[36545]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:38.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:37 smithi186 ceph-mon[36412]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:40.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:39 smithi156 ceph-mon[32025]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:40.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:40.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:39 smithi192 ceph-mon[36545]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:40.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:40.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:39 smithi186 ceph-mon[36412]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:40.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:40.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:41.836 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:41.836 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:41.836 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:41.836 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:41.836 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:41.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.473 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.472+0000 7fcc6dffb700 1 -- 172.21.15.156:0/4140884318 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc70063980 con 0x7fcc5c03e750 2024-05-26T11:54:42.474 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 0 on host 'smithi156' 2024-05-26T11:54:42.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 msgr2=0x7fcc5c040c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:42.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 0x7fcc5c040c10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcc70103500 tx=0x7fcc60007400 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 msgr2=0x7fcc7019fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcc68007f10 tx=0x7fcc6800d3b0 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 shutdown_connections 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcc5c03e750 0x7fcc5c040c10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcc70104040 0x7fcc7019fb80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcc701024a0 0x7fcc701985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 --2- 172.21.15.156:0/4140884318 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc701036a0 0x7fcc70198b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 >> 172.21.15.156:0/4140884318 conn(0x7fcc700fda40 msgr2=0x7fcc70106ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 shutdown_connections 2024-05-26T11:54:42.476 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:54:42.475+0000 7fcc7633e700 1 -- 172.21.15.156:0/4140884318 wait complete. 2024-05-26T11:54:42.722 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:42 smithi156 ceph-mon[32025]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:42.723 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:42 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.723 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:42 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:42.723 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:42 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:42 smithi186 ceph-mon[36412]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:42 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:42 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:42.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:42 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:43.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:42 smithi192 ceph-mon[36545]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:43.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:42 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:43.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:42 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:54:43.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:42 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:43.152 DEBUG:teuthology.orchestra.run.smithi156:osd.0> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.0.service 2024-05-26T11:54:43.154 INFO:tasks.cephadm:Deploying osd.1 on smithi186 with /dev/vg_nvme/lv_4... 2024-05-26T11:54:43.155 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-26T11:54:43.396 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:54:43.704 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:43 smithi156 ceph-mon[32025]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:43.704 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:43 smithi156 ceph-mon[32025]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:43.704 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:43 smithi156 ceph-mon[32025]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:43.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:43 smithi186 ceph-mon[36412]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:43.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:43 smithi186 ceph-mon[36412]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:43.795 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:43 smithi186 ceph-mon[36412]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:43 smithi192 ceph-mon[36545]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:43 smithi192 ceph-mon[36545]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-26T11:54:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:43 smithi192 ceph-mon[36545]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:44.930 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:44 smithi156 ceph-mon[32025]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-26T11:54:44.930 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:44 smithi156 ceph-mon[32025]: osdmap e6: 1 total, 0 up, 1 in 2024-05-26T11:54:44.930 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:44.930 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:44 smithi156 ceph-mon[32025]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:44.930 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:44 smithi156 ceph-mon[32025]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:44 smithi186 ceph-mon[36412]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-26T11:54:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:44 smithi186 ceph-mon[36412]: osdmap e6: 1 total, 0 up, 1 in 2024-05-26T11:54:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:44 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:44 smithi186 ceph-mon[36412]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:44 smithi186 ceph-mon[36412]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:44 smithi192 ceph-mon[36545]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-26T11:54:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:44 smithi192 ceph-mon[36545]: osdmap e6: 1 total, 0 up, 1 in 2024-05-26T11:54:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:44 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:44 smithi192 ceph-mon[36545]: from='osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:44 smithi192 ceph-mon[36545]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113385 s, 925 MB/s 2024-05-26T11:54:45.078 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-26T11:54:45.295 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:54:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0[44160]: 2024-05-26T11:54:45.041+0000 7f4efef1d700 -1 osd.0 0 waiting for initial osdmap 2024-05-26T11:54:45.295 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:54:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0[44160]: 2024-05-26T11:54:45.045+0000 7f4ef6d08700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:54:45.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:54:45.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: osdmap e7: 1 total, 0 up, 1 in 2024-05-26T11:54:45.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:45.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:45.649 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:45.650 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:45 smithi186 ceph-mon[36412]: from='osd.0 ' entity='osd.0' 2024-05-26T11:54:45.676 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_4 2024-05-26T11:54:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:54:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: osdmap e7: 1 total, 0 up, 1 in 2024-05-26T11:54:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:45 smithi156 ceph-mon[32025]: from='osd.0 ' entity='osd.0' 2024-05-26T11:54:45.915 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: osdmap e7: 1 total, 0 up, 1 in 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-26T11:54:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:45 smithi192 ceph-mon[36545]: from='osd.0 ' entity='osd.0' 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] boot 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: osdmap e8: 1 total, 1 up, 1 in 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi156 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi156 to 17088M 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.947 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:46.948 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:46.948 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] boot 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: osdmap e8: 1 total, 1 up, 1 in 2024-05-26T11:54:46.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi156 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi156 to 17088M 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:46.974 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:47.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:54:47.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: osd.0 [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] boot 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: osdmap e8: 1 total, 1 up, 1 in 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi156 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi156 to 17088M 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:54:47.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 -- 172.21.15.186:0/3682113526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc103690 msgr2=0x7f8fbc103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 --2- 172.21.15.186:0/3682113526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc103690 0x7f8fbc103af0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8fa40099c0 tx=0x7f8fa4009cd0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 -- 172.21.15.186:0/3682113526 shutdown_connections 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 --2- 172.21.15.186:0/3682113526 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc104030 0x7f8fbc10a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 --2- 172.21.15.186:0/3682113526 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 --2- 172.21.15.186:0/3682113526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc103690 0x7f8fbc103af0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.423 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 -- 172.21.15.186:0/3682113526 >> 172.21.15.186:0/3682113526 conn(0x7f8fbc0fda70 msgr2=0x7f8fbc0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 -- 172.21.15.186:0/3682113526 shutdown_connections 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.420+0000 7f8fc1443700 1 -- 172.21.15.186:0/3682113526 wait complete. 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 Processor -- start 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 -- start start 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc106420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc103690 0x7f8fbc104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10e020 con 0x7f8fbc102490 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10dea0 con 0x7f8fbc104030 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fc1443700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8fbc10e1a0 con 0x7f8fbc103690 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fbb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fbb7fe700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:40244/0 (socket says 172.21.15.186:40244) 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fbb7fe700 1 -- 172.21.15.186:0/4258823083 learned_addr learned my addr 172.21.15.186:0/4258823083 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.421+0000 7f8fbaffd700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc106420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:47.424 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fb27fc700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc103690 0x7f8fbc104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 -- 172.21.15.186:0/4258823083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc103690 msgr2=0x7f8fbc104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc103690 0x7f8fbc104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 -- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 msgr2=0x7f8fbc106420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc106420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 -- 172.21.15.186:0/4258823083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fbc105860 con 0x7f8fbc104030 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbaffd700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fbb7fe700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400ada0 tx=0x7f8fb400c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4017020 con 0x7f8fbc104030 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8fb400ce30 con 0x7f8fbc104030 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fbc1a39f0 con 0x7f8fbc104030 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fb4005350 con 0x7f8fbc104030 2024-05-26T11:54:47.425 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.422+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fbc1a3eb0 con 0x7f8fbc104030 2024-05-26T11:54:47.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.423+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fbc1a4080 con 0x7f8fbc104030 2024-05-26T11:54:47.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.424+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f8fb400c980 con 0x7f8fbc104030 2024-05-26T11:54:47.426 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.425+0000 7f8fb8ff9700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 0x7f8fa80b4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:54:47.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.425+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1767+0+0 (secure 0 0 0) 0x7f8fb40dfec0 con 0x7f8fbc104030 2024-05-26T11:54:47.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.425+0000 7f8fbaffd700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 0x7f8fa80b4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:54:47.427 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.426+0000 7f8fbaffd700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 0x7f8fa80b4cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1034f0 tx=0x7f8fac009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:54:47.430 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.430+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8fb4021020 con 0x7f8fbc104030 2024-05-26T11:54:47.643 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:54:47.642+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8fbc1a4250 con 0x7f8fa80b27f0 2024-05-26T11:54:47.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:47 smithi186 ceph-mon[36412]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:48.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:47 smithi156 ceph-mon[32025]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:48.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:47 smithi192 ceph-mon[36545]: pgmap v48: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:48.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:48 smithi186 ceph-mon[36412]: osdmap e9: 1 total, 1 up, 1 in 2024-05-26T11:54:48.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:48 smithi186 ceph-mon[36412]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:48.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:48.890 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:48.890 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:49.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:48 smithi156 ceph-mon[32025]: osdmap e9: 1 total, 1 up, 1 in 2024-05-26T11:54:49.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:48 smithi156 ceph-mon[32025]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:49.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:49.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:49.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:49.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:48 smithi192 ceph-mon[36545]: osdmap e9: 1 total, 1 up, 1 in 2024-05-26T11:54:49.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:48 smithi192 ceph-mon[36545]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:54:49.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:54:49.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:54:49.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:49 smithi186 ceph-mon[36412]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:50.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:49 smithi156 ceph-mon[32025]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:50.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:49 smithi192 ceph-mon[36545]: pgmap v50: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:50 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/1426754761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:50 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:50 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]': finished 2024-05-26T11:54:50.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:50 smithi186 ceph-mon[36412]: osdmap e10: 2 total, 1 up, 2 in 2024-05-26T11:54:50.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:54:51.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:50 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/1426754761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:51.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:50 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:51.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:50 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]': finished 2024-05-26T11:54:51.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:50 smithi156 ceph-mon[32025]: osdmap e10: 2 total, 1 up, 2 in 2024-05-26T11:54:51.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:54:51.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:50 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/1426754761' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:51.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:50 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]: dispatch 2024-05-26T11:54:51.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:50 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44233737-91d9-46b7-b9d5-fc82f5ac39ca"}]': finished 2024-05-26T11:54:51.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:50 smithi192 ceph-mon[36545]: osdmap e10: 2 total, 1 up, 2 in 2024-05-26T11:54:51.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:54:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:51 smithi186 ceph-mon[36412]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:51 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/1388608109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:52.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:51 smithi156 ceph-mon[32025]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:52.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:51 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/1388608109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:51 smithi192 ceph-mon[36545]: pgmap v52: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:51 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/1388608109' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:54:54.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:53 smithi156 ceph-mon[32025]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:54.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:53 smithi192 ceph-mon[36545]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:54.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:53 smithi186 ceph-mon[36412]: pgmap v53: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:56.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:55 smithi156 ceph-mon[32025]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:55 smithi192 ceph-mon[36545]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:56.111 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:55 smithi186 ceph-mon[36412]: pgmap v54: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:58.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:57 smithi156 ceph-mon[32025]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:58.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-26T11:54:58.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:58.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:57 smithi156 ceph-mon[32025]: Deploying daemon osd.1 on smithi186 2024-05-26T11:54:58.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:57 smithi192 ceph-mon[36545]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:58.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-26T11:54:58.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:58.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:57 smithi192 ceph-mon[36545]: Deploying daemon osd.1 on smithi186 2024-05-26T11:54:58.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:57 smithi186 ceph-mon[36412]: pgmap v55: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:54:58.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:57 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-26T11:54:58.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:57 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:54:58.136 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:57 smithi186 ceph-mon[36412]: Deploying daemon osd.1 on smithi186 2024-05-26T11:54:59.906 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:54:59 smithi186 ceph-mon[36412]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:00.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:54:59 smithi156 ceph-mon[32025]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:00.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:54:59 smithi192 ceph-mon[36545]: pgmap v56: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:01.897 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:01 smithi186 ceph-mon[36412]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:02.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:01 smithi156 ceph-mon[32025]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:02.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:01 smithi192 ceph-mon[36545]: pgmap v57: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:03 smithi186 ceph-mon[36412]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:03 smithi156 ceph-mon[32025]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:03 smithi192 ceph-mon[36545]: pgmap v58: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:06.256 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.254+0000 7f8fb8ff9700 1 -- 172.21.15.186:0/4258823083 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8fbc1a4250 con 0x7f8fa80b27f0 2024-05-26T11:55:06.256 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 1 on host 'smithi186' 2024-05-26T11:55:06.258 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 msgr2=0x7f8fa80b4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 0x7f8fa80b4cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc1034f0 tx=0x7f8fac009190 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 msgr2=0x7f8fbc104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8fb400ada0 tx=0x7f8fb400c330 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 shutdown_connections 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8fa80b27f0 0x7f8fa80b4cb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc104030 0x7f8fbc104fb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8fbc102490 0x7f8fbc106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 --2- 172.21.15.186:0/4258823083 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8fbc103690 0x7f8fbc104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 >> 172.21.15.186:0/4258823083 conn(0x7f8fbc0fda70 msgr2=0x7f8fbc06bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 shutdown_connections 2024-05-26T11:55:06.259 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:55:06.257+0000 7f8fc1443700 1 -- 172.21.15.186:0/4258823083 wait complete. 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.561 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.562 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.562 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:06.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: pgmap v59: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:06.859 DEBUG:teuthology.orchestra.run.smithi186:osd.1> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.1.service 2024-05-26T11:55:06.861 INFO:tasks.cephadm:Deploying osd.2 on smithi192 with /dev/vg_nvme/lv_4... 2024-05-26T11:55:06.861 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-26T11:55:07.102 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:55:07.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:07 smithi186 ceph-mon[36412]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:07.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:07 smithi186 ceph-mon[36412]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:07.555 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:07 smithi192 ceph-mon[36545]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:07.556 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:07 smithi192 ceph-mon[36545]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:07.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:07 smithi156 ceph-mon[32025]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:07.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:07 smithi156 ceph-mon[32025]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: osdmap e11: 2 total, 1 up, 2 in 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:08.462 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:08 smithi192 ceph-mon[36545]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-26T11:55:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: osdmap e11: 2 total, 1 up, 2 in 2024-05-26T11:55:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:08.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:08 smithi186 ceph-mon[36412]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:08.727 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1[40951]: 2024-05-26T11:55:08.357+0000 7f0db22b7700 -1 osd.1 0 waiting for initial osdmap 2024-05-26T11:55:08.727 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1[40951]: 2024-05-26T11:55:08.359+0000 7f0dae0aa700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:55:08.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:08.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-26T11:55:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: osdmap e11: 2 total, 1 up, 2 in 2024-05-26T11:55:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: from='osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:08 smithi156 ceph-mon[32025]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111354 s, 942 MB/s 2024-05-26T11:55:09.036 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-05-26T11:55:09.467 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:09 smithi186 ceph-mon[36412]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:55:09.467 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:09 smithi186 ceph-mon[36412]: osdmap e12: 2 total, 1 up, 2 in 2024-05-26T11:55:09.467 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.467 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.608 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:09 smithi192 ceph-mon[36545]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:55:09.608 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:09 smithi192 ceph-mon[36545]: osdmap e12: 2 total, 1 up, 2 in 2024-05-26T11:55:09.608 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.608 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.636 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_4 2024-05-26T11:55:09.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:09 smithi156 ceph-mon[32025]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:55:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:09 smithi156 ceph-mon[32025]: osdmap e12: 2 total, 1 up, 2 in 2024-05-26T11:55:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:09.881 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] boot 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: osdmap e13: 2 total, 2 up, 2 in 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:10.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:10.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] boot 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: osdmap e13: 2 total, 2 up, 2 in 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:10.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:10.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:10.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: pgmap v63: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: osd.1 [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] boot 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: osdmap e13: 2 total, 2 up, 2 in 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:10.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:10.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:11.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 -- 172.21.15.192:0/3845906407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c104040 msgr2=0x7fb73c10a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:11.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 --2- 172.21.15.192:0/3845906407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c104040 0x7fb73c10a8f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb72c009990 tx=0x7fb72c009ca0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.482 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 -- 172.21.15.192:0/3845906407 shutdown_connections 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 --2- 172.21.15.192:0/3845906407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c104040 0x7fb73c10a8f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 --2- 172.21.15.192:0/3845906407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c1024a0 0x7fb73c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 --2- 172.21.15.192:0/3845906407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 -- 172.21.15.192:0/3845906407 >> 172.21.15.192:0/3845906407 conn(0x7fb73c0fda40 msgr2=0x7fb73c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 -- 172.21.15.192:0/3845906407 shutdown_connections 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.479+0000 7fb74110e700 1 -- 172.21.15.192:0/3845906407 wait complete. 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 Processor -- start 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 -- start start 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:11.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c104040 0x7fb73c1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c10df90 con 0x7fb73c1036a0 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c10de10 con 0x7fb73c104040 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb74110e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb73c10e110 con 0x7fb73c1024a0 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb73ad9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb73ad9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:42526/0 (socket says 172.21.15.192:42526) 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb73ad9d700 1 -- 172.21.15.192:0/3754520864 learned_addr learned my addr 172.21.15.192:0/3754520864 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.480+0000 7fb73b59e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c104040 0x7fb73c1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73a59c700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 -- 172.21.15.192:0/3754520864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c104040 msgr2=0x7fb73c1a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c104040 0x7fb73c1a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 -- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 msgr2=0x7fb73c199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:11.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 -- 172.21.15.192:0/3754520864 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb73c1048c0 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73a59c700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb73ad9d700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb734007f40 tx=0x7fb73400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb73400ddd0 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb734018070 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7340125b0 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb73c104bb0 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.481+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb73c1050f0 con 0x7fb73c1024a0 2024-05-26T11:55:11.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.482+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb73c06a230 con 0x7fb73c1024a0 2024-05-26T11:55:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.483+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7fb7340133f0 con 0x7fb73c1024a0 2024-05-26T11:55:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.484+0000 7fb733fff700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 0x7fb7280b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.484+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7fb7340e0ad0 con 0x7fb73c1024a0 2024-05-26T11:55:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.484+0000 7fb73a59c700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 0x7fb7280b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:11.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.485+0000 7fb73a59c700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 0x7fb7280b4d30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb724005fd0 tx=0x7fb724005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:55:11.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.488+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb734010070 con 0x7fb73c1024a0 2024-05-26T11:55:11.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:11.696+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fb73c0ffbf0 con 0x7fb7280b2870 2024-05-26T11:55:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:11 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi186 2024-05-26T11:55:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:11 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-26T11:55:11.697 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:11 smithi192 ceph-mon[36545]: osdmap e14: 2 total, 2 up, 2 in 2024-05-26T11:55:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:11 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi186 2024-05-26T11:55:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:11 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-26T11:55:11.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:11 smithi186 ceph-mon[36412]: osdmap e14: 2 total, 2 up, 2 in 2024-05-26T11:55:11.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:11 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi186 2024-05-26T11:55:11.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:11 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi186 to 17088M 2024-05-26T11:55:11.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:11 smithi156 ceph-mon[32025]: osdmap e14: 2 total, 2 up, 2 in 2024-05-26T11:55:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:12 smithi192 ceph-mon[36545]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:12 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:12 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:12 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:12 smithi186 ceph-mon[36412]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:12 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:12 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:12 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:12.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:12 smithi156 ceph-mon[32025]: pgmap v66: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:12.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:12 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:12.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:12 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:12.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:12 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:13.522 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:13 smithi192 ceph-mon[36545]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:13.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:13 smithi186 ceph-mon[36412]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:13.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:13 smithi156 ceph-mon[32025]: from='client.24154 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/2035137058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]': finished 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: osdmap e15: 3 total, 2 up, 3 in 2024-05-26T11:55:14.700 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:14 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/2035137058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]': finished 2024-05-26T11:55:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: osdmap e15: 3 total, 2 up, 3 in 2024-05-26T11:55:14.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:14 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/2035137058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]: dispatch 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "50cc46b8-738c-4301-b70c-a20bbd1029fe"}]': finished 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: osdmap e15: 3 total, 2 up, 3 in 2024-05-26T11:55:14.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:14 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:15 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/2036951445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:15.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:15 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/2036951445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:15.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:15 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/2036951445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:16.723 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:16 smithi192 ceph-mon[36545]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:16.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:16 smithi186 ceph-mon[36412]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:16.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:16 smithi156 ceph-mon[32025]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:18 smithi192 ceph-mon[36545]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:18 smithi186 ceph-mon[36412]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:18.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:18 smithi156 ceph-mon[32025]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:20.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:20 smithi186 ceph-mon[36412]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:20.787 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:20 smithi192 ceph-mon[36545]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:20.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:20 smithi156 ceph-mon[32025]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:21.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-26T11:55:21.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:21.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-26T11:55:21.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:21.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-26T11:55:21.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:22 smithi192 ceph-mon[36545]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:22 smithi192 ceph-mon[36545]: Deploying daemon osd.2 on smithi192 2024-05-26T11:55:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:22 smithi186 ceph-mon[36412]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:22 smithi186 ceph-mon[36412]: Deploying daemon osd.2 on smithi192 2024-05-26T11:55:22.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:22 smithi156 ceph-mon[32025]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:22.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:22 smithi156 ceph-mon[32025]: Deploying daemon osd.2 on smithi192 2024-05-26T11:55:24.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:24 smithi192 ceph-mon[36545]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:24 smithi186 ceph-mon[36412]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:24.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:24 smithi156 ceph-mon[32025]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:26.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:26 smithi186 ceph-mon[36412]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:26.790 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:26 smithi192 ceph-mon[36545]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:26.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:26 smithi156 ceph-mon[32025]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:27.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:27.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:27 smithi192 ceph-mon[36545]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:27.966 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.966 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.966 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:27.966 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:27 smithi156 ceph-mon[32025]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:27 smithi186 ceph-mon[36412]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:29.971 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:29 smithi192 ceph-mon[36545]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:30.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:29 smithi156 ceph-mon[32025]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:30.059 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.059+0000 7fb733fff700 1 -- 172.21.15.192:0/3754520864 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb73c0ffbf0 con 0x7fb7280b2870 2024-05-26T11:55:30.060 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 2 on host 'smithi192' 2024-05-26T11:55:30.062 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 msgr2=0x7fb7280b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:30.062 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 0x7fb7280b4d30 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fb724005fd0 tx=0x7fb724005e30 comp rx=0 tx=0).stop 2024-05-26T11:55:30.062 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 msgr2=0x7fb73c1996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb734007f40 tx=0x7fb73400d3b0 comp rx=0 tx=0).stop 2024-05-26T11:55:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 shutdown_connections 2024-05-26T11:55:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb7280b2870 0x7fb7280b4d30 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb73c104040 0x7fb73c1a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:30.065 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb73c1024a0 0x7fb73c1996b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 --2- 172.21.15.192:0/3754520864 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb73c1036a0 0x7fb73c199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 >> 172.21.15.192:0/3754520864 conn(0x7fb73c0fda40 msgr2=0x7fb73c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:55:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.061+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 shutdown_connections 2024-05-26T11:55:30.066 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:55:30.062+0000 7fb74110e700 1 -- 172.21.15.192:0/3754520864 wait complete. 2024-05-26T11:55:30.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:29 smithi186 ceph-mon[36412]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:30.698 DEBUG:teuthology.orchestra.run.smithi192:osd.2> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.2.service 2024-05-26T11:55:30.700 INFO:tasks.cephadm:Deploying osd.3 on smithi156 with /dev/vg_nvme/lv_3... 2024-05-26T11:55:30.701 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-26T11:55:30.947 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:31.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:31 smithi156 ceph-mon[32025]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:31.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:31 smithi192 ceph-mon[36545]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:31.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:31.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:31.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:31 smithi186 ceph-mon[36412]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-26T11:55:32.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:32.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-26T11:55:32.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: osdmap e16: 3 total, 2 up, 3 in 2024-05-26T11:55:32.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:32.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:32 smithi156 ceph-mon[32025]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:32.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-26T11:55:32.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: osdmap e16: 3 total, 2 up, 3 in 2024-05-26T11:55:32.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:32.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:32 smithi186 ceph-mon[36412]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:32.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-26T11:55:32.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: osdmap e16: 3 total, 2 up, 3 in 2024-05-26T11:55:32.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:32.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: from='osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:32 smithi192 ceph-mon[36545]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:55:32.573 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:32 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2[41782]: 2024-05-26T11:55:32.132+0000 7f0749e5d700 -1 osd.2 0 waiting for initial osdmap 2024-05-26T11:55:32.573 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:32 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2[41782]: 2024-05-26T11:55:32.136+0000 7f0745c50700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:55:32.940 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-26T11:55:32.940 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-26T11:55:32.941 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-05-26T11:55:32.941 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-05-26T11:55:32.941 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00861111 s, 1.2 GB/s 2024-05-26T11:55:32.941 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-05-26T11:55:33.233 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:33 smithi192 ceph-mon[36545]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:55:33.233 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:33 smithi192 ceph-mon[36545]: osdmap e17: 3 total, 2 up, 3 in 2024-05-26T11:55:33.233 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.233 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:33 smithi186 ceph-mon[36412]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:55:33.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:33 smithi186 ceph-mon[36412]: osdmap e17: 3 total, 2 up, 3 in 2024-05-26T11:55:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:33 smithi156 ceph-mon[32025]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:55:33.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:33 smithi156 ceph-mon[32025]: osdmap e17: 3 total, 2 up, 3 in 2024-05-26T11:55:33.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:33.665 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_3 2024-05-26T11:55:33.908 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:55:34.287 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:55:34.287 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] boot 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: osdmap e18: 3 total, 3 up, 3 in 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi192 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.288 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.289 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:34.289 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.289 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.289 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:55:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] boot 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: osdmap e18: 3 total, 3 up, 3 in 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi192 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:34.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: osd.2 [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] boot 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: osdmap e18: 3 total, 3 up, 3 in 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-26T11:55:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi192 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:34.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:35.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:35 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi192 to 17088M 2024-05-26T11:55:35.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:35 smithi156 ceph-mon[32025]: osdmap e19: 3 total, 3 up, 3 in 2024-05-26T11:55:35.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:35.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:35 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi192 to 17088M 2024-05-26T11:55:35.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:35 smithi186 ceph-mon[36412]: osdmap e19: 3 total, 3 up, 3 in 2024-05-26T11:55:35.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:35 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:35.544 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- 172.21.15.156:0/3062079031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834104790 msgr2=0x7f4834104bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 --2- 172.21.15.156:0/3062079031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834104790 0x7f4834104bf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f48280099c0 tx=0x7f4828009cd0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- 172.21.15.156:0/3062079031 shutdown_connections 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 --2- 172.21.15.156:0/3062079031 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834105130 0x7f483410b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 --2- 172.21.15.156:0/3062079031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834103590 0x7f48341039b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 --2- 172.21.15.156:0/3062079031 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834104790 0x7f4834104bf0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- 172.21.15.156:0/3062079031 >> 172.21.15.156:0/3062079031 conn(0x7f48340feb30 msgr2=0x7f4834100f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- 172.21.15.156:0/3062079031 shutdown_connections 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- 172.21.15.156:0/3062079031 wait complete. 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 Processor -- start 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.540+0000 7f4839f59700 1 -- start start 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834103590 0x7f48341996c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:35.545 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834104790 0x7f4834199c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f483410f000 con 0x7f4834105130 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f483410ee80 con 0x7f4834104790 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f483410f180 con 0x7f4834103590 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:60462/0 (socket says 172.21.15.156:60462) 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 -- 172.21.15.156:0/3207807135 learned_addr learned my addr 172.21.15.156:0/3207807135 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f48337fe700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834103590 0x7f48341996c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4832ffd700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834104790 0x7f4834199c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 -- 172.21.15.156:0/3207807135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834103590 msgr2=0x7f48341996c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:35.546 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834103590 0x7f48341996c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 -- 172.21.15.156:0/3207807135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834104790 msgr2=0x7f4834199c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834104790 0x7f4834199c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 -- 172.21.15.156:0/3207807135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48341a1320 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4833fff700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f482c007f10 tx=0x7f482c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c00ddd0 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f482c004020 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48341a1610 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.541+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c015660 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.542+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48341a1b00 con 0x7f4834105130 2024-05-26T11:55:35.547 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.542+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f483406a230 con 0x7f4834105130 2024-05-26T11:55:35.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.544+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 15) ==== 149597+0+0 (secure 0 0 0) 0x7f482c0044f0 con 0x7f4834105130 2024-05-26T11:55:35.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.545+0000 7f4830ff9700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 0x7f481c0b4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:55:35.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.545+0000 7f48337fe700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 0x7f481c0b4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:55:35.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.545+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3226+0+0 (secure 0 0 0) 0x7f482c0e1020 con 0x7f4834105130 2024-05-26T11:55:35.548 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.546+0000 7f48337fe700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 0x7f481c0b4d30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48341045f0 tx=0x7f482400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:55:35.549 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.549+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f482c010070 con 0x7f4834105130 2024-05-26T11:55:35.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:35 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi192 to 17088M 2024-05-26T11:55:35.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:35 smithi192 ceph-mon[36545]: osdmap e19: 3 total, 3 up, 3 in 2024-05-26T11:55:35.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:35 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:35.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:35.763+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f4834063980 con 0x7f481c0b2870 2024-05-26T11:55:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: osdmap e20: 3 total, 3 up, 3 in 2024-05-26T11:55:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:36.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:36.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:36.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:36.496 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: osdmap e20: 3 total, 3 up, 3 in 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:36.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: pgmap v83: 0 pgs: ; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: osdmap e20: 3 total, 3 up, 3 in 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:55:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:36.794 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48539]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:36.794 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48539]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48539]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48539]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48542]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48542]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48542]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:36.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48542]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45391]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45391]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45391]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45391]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45394]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45394]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45394]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45394]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45385]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45385]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45385]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45385]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45388]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45388]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45388]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.073 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 11:55:36 smithi192 sudo[45388]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.155 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48545]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:37.155 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48545]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.155 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48545]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.155 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48545]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48548]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48548]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48548]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:36 smithi156 sudo[48548]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45048]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45048]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45048]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45048]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45051]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45051]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.156 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45051]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.157 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45051]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45042]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45042]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45042]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45042]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45045]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45045]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45045]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-26T11:55:37.157 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 11:55:36 smithi186 sudo[45045]: pam_unix(sudo:session): session closed for user root 2024-05-26T11:55:37.406 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: osdmap e21: 3 total, 3 up, 3 in 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.407 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.408 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:37 smithi156 ceph-mon[32025]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:37.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: osdmap e21: 3 total, 3 up, 3 in 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.478 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:37 smithi186 ceph-mon[36412]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: osdmap e21: 3 total, 3 up, 3 in 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:37.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-26T11:55:37.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:37 smithi192 ceph-mon[36545]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-26T11:55:37.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:37.732+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f482c0a5cf0 con 0x7f4834105130 2024-05-26T11:55:38.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:38 smithi186 ceph-mon[36412]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:38.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:38 smithi186 ceph-mon[36412]: osdmap e22: 3 total, 3 up, 3 in 2024-05-26T11:55:38.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:38 smithi186 ceph-mon[36412]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-26T11:55:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:38 smithi156 ceph-mon[32025]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:38 smithi156 ceph-mon[32025]: osdmap e22: 3 total, 3 up, 3 in 2024-05-26T11:55:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:38 smithi156 ceph-mon[32025]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-26T11:55:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:38 smithi192 ceph-mon[36545]: pgmap v86: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:38 smithi192 ceph-mon[36545]: osdmap e22: 3 total, 3 up, 3 in 2024-05-26T11:55:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:38 smithi192 ceph-mon[36545]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-26T11:55:39.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:39 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]: dispatch 2024-05-26T11:55:39.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:39 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]': finished 2024-05-26T11:55:39.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:39 smithi186 ceph-mon[36412]: osdmap e23: 4 total, 3 up, 4 in 2024-05-26T11:55:39.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:39.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:39 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/954937434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:39.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:39 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]: dispatch 2024-05-26T11:55:39.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:39 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]': finished 2024-05-26T11:55:39.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:39 smithi156 ceph-mon[32025]: osdmap e23: 4 total, 3 up, 4 in 2024-05-26T11:55:39.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:39.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:39 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/954937434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:39.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:39 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]: dispatch 2024-05-26T11:55:39.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:39 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/4017857984' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "31983ecf-dad2-4a4b-b72a-d24f0f8148ff"}]': finished 2024-05-26T11:55:39.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:39 smithi192 ceph-mon[36545]: osdmap e23: 4 total, 3 up, 4 in 2024-05-26T11:55:39.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:39.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:39 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/954937434' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:55:40.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:40 smithi156 ceph-mon[32025]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:40 smithi192 ceph-mon[36545]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:40 smithi186 ceph-mon[36412]: pgmap v89: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:42.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:42 smithi156 ceph-mon[32025]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:42.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:42 smithi192 ceph-mon[36545]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:42 smithi186 ceph-mon[36412]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:44.750 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:44 smithi156 ceph-mon[32025]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:44.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:44 smithi192 ceph-mon[36545]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:44 smithi186 ceph-mon[36412]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:46.778 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:46 smithi186 ceph-mon[36412]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:46.779 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-26T11:55:46.779 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:46.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:46 smithi156 ceph-mon[32025]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:46.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-26T11:55:46.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:46.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:46 smithi192 ceph-mon[36545]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-26T11:55:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:47.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:47 smithi156 ceph-mon[32025]: Deploying daemon osd.3 on smithi156 2024-05-26T11:55:47.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:47 smithi192 ceph-mon[36545]: Deploying daemon osd.3 on smithi156 2024-05-26T11:55:47.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:47 smithi186 ceph-mon[36412]: Deploying daemon osd.3 on smithi156 2024-05-26T11:55:48.769 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:48 smithi156 ceph-mon[32025]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:48.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:48 smithi192 ceph-mon[36545]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:48.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:48 smithi186 ceph-mon[36412]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:50.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:50 smithi156 ceph-mon[32025]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:50 smithi192 ceph-mon[36545]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:50 smithi186 ceph-mon[36412]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:52.726 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:52 smithi156 ceph-mon[32025]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:52.726 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:52 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.726 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:52 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.726 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:52 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:52 smithi192 ceph-mon[36545]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:52 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:52 smithi186 ceph-mon[36412]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:52 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:52 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:52 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:54.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:54 smithi156 ceph-mon[32025]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:54.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:54 smithi192 ceph-mon[36545]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:54.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:54 smithi186 ceph-mon[36412]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:55.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.579+0000 7f4830ff9700 1 -- 172.21.15.156:0/3207807135 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4834063980 con 0x7f481c0b2870 2024-05-26T11:55:55.580 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 3 on host 'smithi156' 2024-05-26T11:55:55.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 msgr2=0x7f481c0b4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:55.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 0x7f481c0b4d30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f48341045f0 tx=0x7f482400a380 comp rx=0 tx=0).stop 2024-05-26T11:55:55.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 msgr2=0x7f48341a0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:55:55.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f482c007f10 tx=0x7f482c00d3b0 comp rx=0 tx=0).stop 2024-05-26T11:55:55.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 shutdown_connections 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f481c0b2870 0x7f481c0b4d30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4834105130 0x7f48341a0c80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4834103590 0x7f48341996c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.581+0000 7f4839f59700 1 --2- 172.21.15.156:0/3207807135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4834104790 0x7f4834199c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.582+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 >> 172.21.15.156:0/3207807135 conn(0x7f48340feb30 msgr2=0x7f4834107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.582+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 shutdown_connections 2024-05-26T11:55:55.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:55:55.582+0000 7f4839f59700 1 -- 172.21.15.156:0/3207807135 wait complete. 2024-05-26T11:55:55.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:55 smithi156 ceph-mon[32025]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-26T11:55:55.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:55 smithi186 ceph-mon[36412]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:55:56.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:56.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:55 smithi192 ceph-mon[36545]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-26T11:55:56.271 DEBUG:teuthology.orchestra.run.smithi156:osd.3> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.3.service 2024-05-26T11:55:56.274 INFO:tasks.cephadm:Deploying osd.4 on smithi186 with /dev/vg_nvme/lv_3... 2024-05-26T11:55:56.274 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-26T11:55:56.518 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:55:57.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:57 smithi156 ceph-mon[32025]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-26T11:55:57.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:57 smithi156 ceph-mon[32025]: osdmap e24: 4 total, 3 up, 4 in 2024-05-26T11:55:57.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:57 smithi156 ceph-mon[32025]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:55:57.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:57 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:57.796 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 11:55:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T11:55:57.561+0000 7fc81e2af700 -1 osd.3 0 waiting for initial osdmap 2024-05-26T11:55:57.796 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 11:55:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T11:55:57.566+0000 7fc81a0a2700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:55:57.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:57 smithi192 ceph-mon[36545]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-26T11:55:57.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:57 smithi192 ceph-mon[36545]: osdmap e24: 4 total, 3 up, 4 in 2024-05-26T11:55:57.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:57 smithi192 ceph-mon[36545]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:55:57.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:57 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:57.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:57 smithi186 ceph-mon[36412]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-26T11:55:57.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:57 smithi186 ceph-mon[36412]: osdmap e24: 4 total, 3 up, 4 in 2024-05-26T11:55:57.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:57 smithi186 ceph-mon[36412]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:55:57.830 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:57 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:58.548 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-26T11:55:58.549 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-26T11:55:58.549 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-26T11:55:58.549 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-26T11:55:58.549 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0138498 s, 757 MB/s 2024-05-26T11:55:58.549 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-26T11:55:58.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:58 smithi186 ceph-mon[36412]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:58.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:58 smithi186 ceph-mon[36412]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:55:58.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:58 smithi186 ceph-mon[36412]: osdmap e25: 4 total, 3 up, 4 in 2024-05-26T11:55:58.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:58.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:58.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:58 smithi192 ceph-mon[36545]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:58.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:58 smithi192 ceph-mon[36545]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:55:58.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:58 smithi192 ceph-mon[36545]: osdmap e25: 4 total, 3 up, 4 in 2024-05-26T11:55:58.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:58.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.008 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:58 smithi156 ceph-mon[32025]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:55:59.008 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:58 smithi156 ceph-mon[32025]: from='osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:55:59.008 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:58 smithi156 ceph-mon[32025]: osdmap e25: 4 total, 3 up, 4 in 2024-05-26T11:55:59.008 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.008 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.272 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_3 2024-05-26T11:55:59.511 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:55:59.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] boot 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: osdmap e26: 4 total, 4 up, 4 in 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.796 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:59.796 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:59.796 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:55:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:55:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:55:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] boot 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: osdmap e26: 4 total, 4 up, 4 in 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:59.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:59.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:55:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.873 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:55:59.874 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:55:59.874 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.874 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: osd.3 [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] boot 2024-05-26T11:55:59.874 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: osdmap e26: 4 total, 4 up, 4 in 2024-05-26T11:55:59.874 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:55:59.875 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:55:59.876 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:55:59.876 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:55:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:00.938 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:00 smithi186 ceph-mon[36412]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:56:00.939 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:00 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi156 2024-05-26T11:56:00.939 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:00 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi156 to 8544M 2024-05-26T11:56:00.939 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:00 smithi186 ceph-mon[36412]: osdmap e27: 4 total, 4 up, 4 in 2024-05-26T11:56:01.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:00 smithi156 ceph-mon[32025]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:56:01.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:00 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi156 2024-05-26T11:56:01.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:00 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi156 to 8544M 2024-05-26T11:56:01.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:00 smithi156 ceph-mon[32025]: osdmap e27: 4 total, 4 up, 4 in 2024-05-26T11:56:01.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:00 smithi192 ceph-mon[36545]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-05-26T11:56:01.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:00 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi156 2024-05-26T11:56:01.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:00 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi156 to 8544M 2024-05-26T11:56:01.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:00 smithi192 ceph-mon[36545]: osdmap e27: 4 total, 4 up, 4 in 2024-05-26T11:56:01.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.176+0000 7ff1fc777700 1 -- 172.21.15.186:0/2378016755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 msgr2=0x7ff1f4101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:01.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.176+0000 7ff1fc777700 1 --2- 172.21.15.186:0/2378016755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f4101a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff1e4009990 tx=0x7ff1e4009ca0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- 172.21.15.186:0/2378016755 shutdown_connections 2024-05-26T11:56:01.179 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- 172.21.15.186:0/2378016755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f410bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- 172.21.15.186:0/2378016755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 0x7ff1f4104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- 172.21.15.186:0/2378016755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f4101a90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- 172.21.15.186:0/2378016755 >> 172.21.15.186:0/2378016755 conn(0x7ff1f40fd1f0 msgr2=0x7ff1f40ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- 172.21.15.186:0/2378016755 shutdown_connections 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- 172.21.15.186:0/2378016755 wait complete. 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 Processor -- start 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- start start 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f4102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 0x7ff1f4102e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:01.180 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f410f0c0 con 0x7ff1f4104bb0 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f410ef40 con 0x7ff1f4101630 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fc777700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff1f410f240 con 0x7ff1f4101fd0 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fa513700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fa513700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:33762/0 (socket says 172.21.15.186:33762) 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.177+0000 7ff1fa513700 1 -- 172.21.15.186:0/27665091 learned_addr learned my addr 172.21.15.186:0/27665091 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1f9d12700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f4102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fad14700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 0x7ff1f4102e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 -- 172.21.15.186:0/27665091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 msgr2=0x7ff1f4102900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f4102900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 -- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 msgr2=0x7ff1f4102e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 0x7ff1f4102e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:01.181 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 -- 172.21.15.186:0/27665091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1f41036f0 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1f9d12700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f4102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fa513700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c970 tx=0x7ff1f000cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0004220 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff1f0007d50 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1f41a3890 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1f0003d10 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.178+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1f41a3dd0 con 0x7ff1f4101630 2024-05-26T11:56:01.182 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.179+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1f406a230 con 0x7ff1f4101630 2024-05-26T11:56:01.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.180+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7ff1f00043c0 con 0x7ff1f4101630 2024-05-26T11:56:01.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.181+0000 7ff1eb7fe700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 0x7ff1e00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:01.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.182+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7ff1f00e0f90 con 0x7ff1f4101630 2024-05-26T11:56:01.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.182+0000 7ff1f9d12700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 0x7ff1e00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:01.183 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.182+0000 7ff1f9d12700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 0x7ff1e00b4dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b5c0 tx=0x7ff1e4005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:01.185 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.185+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff1f0014030 con 0x7ff1f4101630 2024-05-26T11:56:01.396 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:01.396+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff1f4063980 con 0x7ff1e00b2900 2024-05-26T11:56:01.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:01 smithi186 ceph-mon[36412]: osdmap e28: 4 total, 4 up, 4 in 2024-05-26T11:56:01.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:01.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:01.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:02.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:01 smithi156 ceph-mon[32025]: osdmap e28: 4 total, 4 up, 4 in 2024-05-26T11:56:02.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:02.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:02.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:01 smithi192 ceph-mon[36545]: osdmap e28: 4 total, 4 up, 4 in 2024-05-26T11:56:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:02.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:02 smithi186 ceph-mon[36412]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:02.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:02 smithi186 ceph-mon[36412]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:03.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:02 smithi156 ceph-mon[32025]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:02 smithi156 ceph-mon[32025]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:03.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:02 smithi192 ceph-mon[36545]: pgmap v105: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:03.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:02 smithi192 ceph-mon[36545]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/1305259619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]': finished 2024-05-26T11:56:04.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: osdmap e29: 5 total, 4 up, 5 in 2024-05-26T11:56:04.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:05.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/1305259619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]': finished 2024-05-26T11:56:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: osdmap e29: 5 total, 4 up, 5 in 2024-05-26T11:56:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/1305259619' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]: dispatch 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3d2c072c-0fde-45be-b531-1179e7af010d"}]': finished 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: osdmap e29: 5 total, 4 up, 5 in 2024-05-26T11:56:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:05.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:05 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/2177193068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:06.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:05 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/2177193068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:06.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:05 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/2177193068' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:06.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:06 smithi186 ceph-mon[36412]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:07.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:06 smithi156 ceph-mon[32025]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:07.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:06 smithi192 ceph-mon[36545]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:07 smithi186 ceph-mon[36412]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:56:08.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:07 smithi156 ceph-mon[32025]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:56:08.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:07 smithi192 ceph-mon[36545]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:56:10.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:09 smithi156 ceph-mon[32025]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-26T11:56:10.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:09 smithi192 ceph-mon[36545]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-26T11:56:10.128 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:09 smithi186 ceph-mon[36412]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 71 KiB/s, 0 objects/s recovering 2024-05-26T11:56:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:11 smithi192 ceph-mon[36545]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-26T11:56:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:11 smithi192 ceph-mon[36545]: Deploying daemon osd.4 on smithi186 2024-05-26T11:56:12.174 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:11 smithi186 ceph-mon[36412]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:12.174 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-26T11:56:12.174 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:12.174 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:11 smithi186 ceph-mon[36412]: Deploying daemon osd.4 on smithi186 2024-05-26T11:56:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:11 smithi156 ceph-mon[32025]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-26T11:56:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:11 smithi156 ceph-mon[32025]: Deploying daemon osd.4 on smithi186 2024-05-26T11:56:14.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:13 smithi192 ceph-mon[36545]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:14.214 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:13 smithi186 ceph-mon[36412]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:14.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:13 smithi156 ceph-mon[32025]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:56:16.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:15 smithi192 ceph-mon[36545]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-26T11:56:16.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:15 smithi186 ceph-mon[36412]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-26T11:56:16.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:15 smithi156 ceph-mon[32025]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 55 KiB/s, 0 objects/s recovering 2024-05-26T11:56:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:18 smithi156 ceph-mon[32025]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-26T11:56:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:18 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:18 smithi192 ceph-mon[36545]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-26T11:56:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:18 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:18 smithi186 ceph-mon[36412]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-26T11:56:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:18 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:20.361 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:20 smithi186 ceph-mon[36412]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:20.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:20 smithi156 ceph-mon[32025]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:20.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:20 smithi192 ceph-mon[36545]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:20.712 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.712+0000 7ff1eb7fe700 1 -- 172.21.15.186:0/27665091 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff1f4063980 con 0x7ff1e00b2900 2024-05-26T11:56:20.713 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 4 on host 'smithi186' 2024-05-26T11:56:20.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.714+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 msgr2=0x7ff1e00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:20.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.714+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 0x7ff1e00b4dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff1e400b5c0 tx=0x7ff1e4005fb0 comp rx=0 tx=0).stop 2024-05-26T11:56:20.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 msgr2=0x7ff1f41042b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:20.715 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff1f000c970 tx=0x7ff1f000cd30 comp rx=0 tx=0).stop 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 shutdown_connections 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff1e00b2900 0x7ff1e00b4dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff1f4104bb0 0x7ff1f4102e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1f4101630 0x7ff1f41042b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 --2- 172.21.15.186:0/27665091 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff1f4101fd0 0x7ff1f4102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 >> 172.21.15.186:0/27665091 conn(0x7ff1f40fd1f0 msgr2=0x7ff1f4107c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 shutdown_connections 2024-05-26T11:56:20.716 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:56:20.715+0000 7ff1fc777700 1 -- 172.21.15.186:0/27665091 wait complete. 2024-05-26T11:56:21.393 DEBUG:teuthology.orchestra.run.smithi186:osd.4> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.4.service 2024-05-26T11:56:21.395 INFO:tasks.cephadm:Deploying osd.5 on smithi192 with /dev/vg_nvme/lv_3... 2024-05-26T11:56:21.395 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-26T11:56:21.639 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:21.761 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:21 smithi192 ceph-mon[36545]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:21.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:21.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:21.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:21 smithi186 ceph-mon[36412]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:22.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:21 smithi156 ceph-mon[32025]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-26T11:56:22.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:22 smithi186 ceph-mon[36412]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-26T11:56:22.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:22 smithi186 ceph-mon[36412]: osdmap e30: 5 total, 4 up, 5 in 2024-05-26T11:56:22.804 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:22.805 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:22 smithi186 ceph-mon[36412]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:22.805 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:22 smithi186 ceph-mon[36412]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:23.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:22 smithi192 ceph-mon[36545]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-26T11:56:23.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:22 smithi192 ceph-mon[36545]: osdmap e30: 5 total, 4 up, 5 in 2024-05-26T11:56:23.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:22 smithi192 ceph-mon[36545]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:23.031 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:22 smithi192 ceph-mon[36545]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:23.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:22 smithi156 ceph-mon[32025]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-26T11:56:23.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:22 smithi156 ceph-mon[32025]: osdmap e30: 5 total, 4 up, 5 in 2024-05-26T11:56:23.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:22 smithi156 ceph-mon[32025]: from='osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:23.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:22 smithi156 ceph-mon[32025]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:56:23.477 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 11:56:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4[47844]: 2024-05-26T11:56:23.187+0000 7fd548f5e700 -1 osd.4 0 waiting for initial osdmap 2024-05-26T11:56:23.477 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 11:56:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4[47844]: 2024-05-26T11:56:23.192+0000 7fd540d49700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:56:23.873 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:23.873 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:56:23.873 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: osdmap e31: 5 total, 4 up, 5 in 2024-05-26T11:56:23.873 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.874 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.874 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:23 smithi192 ceph-mon[36545]: from='osd.4 ' entity='osd.4' 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113042 s, 928 MB/s 2024-05-26T11:56:23.874 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-05-26T11:56:23.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:23.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:56:23.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: osdmap e31: 5 total, 4 up, 5 in 2024-05-26T11:56:23.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:23.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:23 smithi186 ceph-mon[36412]: from='osd.4 ' entity='osd.4' 2024-05-26T11:56:24.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: osdmap e31: 5 total, 4 up, 5 in 2024-05-26T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:24.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:24.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:23 smithi156 ceph-mon[32025]: from='osd.4 ' entity='osd.4' 2024-05-26T11:56:24.556 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_3 2024-05-26T11:56:24.799 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] boot 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: osdmap e32: 5 total, 5 up, 5 in 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi186 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-26T11:56:24.800 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:24.801 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] boot 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: osdmap e32: 5 total, 5 up, 5 in 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi186 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:25.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: osd.4 [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] boot 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: osdmap e32: 5 total, 5 up, 5 in 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi186 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi186 to 8544M 2024-05-26T11:56:25.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:25.228 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:26.505 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- 172.21.15.192:0/3782336735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c1052a0 msgr2=0x7f739c10bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:26.505 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 --2- 172.21.15.192:0/3782336735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c1052a0 0x7f739c10bb50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f739400b0a0 tx=0x7f739400b3b0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- 172.21.15.192:0/3782336735 shutdown_connections 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 --2- 172.21.15.192:0/3782336735 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c1052a0 0x7f739c10bb50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 --2- 172.21.15.192:0/3782336735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c101920 0x7f739c101d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 --2- 172.21.15.192:0/3782336735 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c104900 0x7f739c104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- 172.21.15.192:0/3782336735 >> 172.21.15.192:0/3782336735 conn(0x7f739c0fd280 msgr2=0x7f739c0ff6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- 172.21.15.192:0/3782336735 shutdown_connections 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- 172.21.15.192:0/3782336735 wait complete. 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 Processor -- start 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.503+0000 7f73a3777700 1 -- start start 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c101920 0x7f739c1996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:26.506 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 0x7f739c1a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f739c10f0f0 con 0x7f739c101920 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f739c10ef70 con 0x7f739c1052a0 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f739c10f270 con 0x7f739c104900 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:53646/0 (socket says 172.21.15.192:53646) 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 -- 172.21.15.192:0/3482319446 learned_addr learned my addr 172.21.15.192:0/3482319446 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a1d14700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 0x7f739c1a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a1513700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c101920 0x7f739c1996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 -- 172.21.15.192:0/3482319446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 msgr2=0x7f739c1a0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 0x7f739c1a0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 -- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c101920 msgr2=0x7f739c1996b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:26.507 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c101920 0x7f739c1996b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 -- 172.21.15.192:0/3482319446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739c1a1310 con 0x7f739c104900 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a1d14700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 0x7f739c1a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a0d12700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f738c00d980 tx=0x7f738c00dd40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c016070 con 0x7f739c104900 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f738c010460 con 0x7f739c104900 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f739c1a1600 con 0x7f739c104900 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f738c00f620 con 0x7f739c104900 2024-05-26T11:56:26.508 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.504+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f739c1a1b40 con 0x7f739c104900 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.505+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f739c06a230 con 0x7f739c104900 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.507+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f738c019070 con 0x7f739c104900 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.507+0000 7f73927fc700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 0x7f73880b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.507+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 4101+0+0 (secure 0 0 0) 0x7f738c0e10d0 con 0x7f739c104900 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.508+0000 7f73a1513700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 0x7f73880b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:26.509 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.508+0000 7f73a1513700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 0x7f73880b4d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7398005950 tx=0x7f73980058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:26.512 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.512+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f738c0a9e60 con 0x7f739c104900 2024-05-26T11:56:26.725 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:26.725+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f739c063980 con 0x7f73880b2880 2024-05-26T11:56:26.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:26 smithi186 ceph-mon[36412]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:26.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:26 smithi186 ceph-mon[36412]: osdmap e33: 5 total, 5 up, 5 in 2024-05-26T11:56:27.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:26 smithi156 ceph-mon[32025]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:27.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:26 smithi156 ceph-mon[32025]: osdmap e33: 5 total, 5 up, 5 in 2024-05-26T11:56:27.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:26 smithi192 ceph-mon[36545]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-05-26T11:56:27.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:26 smithi192 ceph-mon[36545]: osdmap e33: 5 total, 5 up, 5 in 2024-05-26T11:56:27.946 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:27.946 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:27.946 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:28.024 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:28.025 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:28.025 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:28 smithi186 ceph-mon[36412]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:28 smithi186 ceph-mon[36412]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:29.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:28 smithi156 ceph-mon[32025]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:29.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:28 smithi156 ceph-mon[32025]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:28 smithi192 ceph-mon[36545]: from='client.24220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:28 smithi192 ceph-mon[36545]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:29.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:29 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/2838192673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:29.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:29 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:29.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:29 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]': finished 2024-05-26T11:56:29.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:29 smithi186 ceph-mon[36412]: osdmap e34: 6 total, 5 up, 6 in 2024-05-26T11:56:29.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:30.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:29 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/2838192673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:30.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:29 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:30.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:29 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]': finished 2024-05-26T11:56:30.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:29 smithi156 ceph-mon[32025]: osdmap e34: 6 total, 5 up, 6 in 2024-05-26T11:56:30.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:29 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/2838192673' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:29 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]: dispatch 2024-05-26T11:56:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:29 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "43f06f8b-265a-4e50-92e9-01a5e2b28d59"}]': finished 2024-05-26T11:56:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:29 smithi192 ceph-mon[36545]: osdmap e34: 6 total, 5 up, 6 in 2024-05-26T11:56:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:30.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:30 smithi186 ceph-mon[36412]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:30.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:30 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/3631133887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:31.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:30 smithi156 ceph-mon[32025]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:31.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:30 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/3631133887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:31.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:30 smithi192 ceph-mon[36545]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 141 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:31.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:30 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/3631133887' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:31.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:31 smithi186 ceph-mon[36412]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:32.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:31 smithi156 ceph-mon[32025]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:32.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:31 smithi192 ceph-mon[36545]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:33 smithi192 ceph-mon[36545]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:34.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:33 smithi186 ceph-mon[36412]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:34.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:33 smithi156 ceph-mon[32025]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:35 smithi192 ceph-mon[36545]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:36.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:35 smithi186 ceph-mon[36412]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:36.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:35 smithi156 ceph-mon[32025]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:37 smithi192 ceph-mon[36545]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-26T11:56:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:37 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:37 smithi192 ceph-mon[36545]: Deploying daemon osd.5 on smithi192 2024-05-26T11:56:38.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:37 smithi186 ceph-mon[36412]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:38.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-26T11:56:38.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:37 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:38.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:37 smithi186 ceph-mon[36412]: Deploying daemon osd.5 on smithi192 2024-05-26T11:56:38.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:37 smithi156 ceph-mon[32025]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:38.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-26T11:56:38.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:37 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:38.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:37 smithi156 ceph-mon[32025]: Deploying daemon osd.5 on smithi192 2024-05-26T11:56:40.028 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:39 smithi192 ceph-mon[36545]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:40.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:39 smithi186 ceph-mon[36412]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:40.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:39 smithi156 ceph-mon[32025]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:41.970 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:41 smithi192 ceph-mon[36545]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:42.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:41 smithi186 ceph-mon[36412]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:42.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:41 smithi156 ceph-mon[32025]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:43.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:43.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:43.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:44.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:44.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:44.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:44 smithi186 ceph-mon[36412]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:45.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:44 smithi156 ceph-mon[32025]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:44 smithi192 ceph-mon[36545]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:46.496 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:46.496 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.496 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.496 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:46.496 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:46.497 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.591+0000 7f73927fc700 1 -- 172.21.15.192:0/3482319446 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f739c063980 con 0x7f73880b2880 2024-05-26T11:56:46.592 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 5 on host 'smithi192' 2024-05-26T11:56:46.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 msgr2=0x7f73880b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 0x7f73880b4d40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f7398005950 tx=0x7f73980058e0 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 msgr2=0x7f739c199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f738c00d980 tx=0x7f738c00dd40 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 shutdown_connections 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73880b2880 0x7f73880b4d40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f739c1052a0 0x7f739c1a0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f739c101920 0x7f739c1996b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 --2- 172.21.15.192:0/3482319446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f739c104900 0x7f739c199bf0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 >> 172.21.15.192:0/3482319446 conn(0x7f739c0fd280 msgr2=0x7f739c107c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 shutdown_connections 2024-05-26T11:56:46.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:56:46.594+0000 7f73a3777700 1 -- 172.21.15.192:0/3482319446 wait complete. 2024-05-26T11:56:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.285 DEBUG:teuthology.orchestra.run.smithi192:osd.5> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.5.service 2024-05-26T11:56:47.288 INFO:tasks.cephadm:Deploying osd.6 on smithi156 with /dev/vg_nvme/lv_2... 2024-05-26T11:56:47.288 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-26T11:56:47.540 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:56:47.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:47 smithi192 ceph-mon[36545]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:47 smithi192 ceph-mon[36545]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:47 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:47 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:47 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:47.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:47 smithi156 ceph-mon[32025]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:47 smithi156 ceph-mon[32025]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:47 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:47 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.663 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:47 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:47.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:47 smithi186 ceph-mon[36412]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:47 smithi186 ceph-mon[36412]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-26T11:56:47.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:47 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:47 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:47.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:47 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:56:48.654 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:48.655 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-26T11:56:48.655 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: osdmap e35: 6 total, 5 up, 6 in 2024-05-26T11:56:48.655 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:48.655 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.655 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:48 smithi192 ceph-mon[36545]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.718 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:48.719 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-26T11:56:48.719 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: osdmap e35: 6 total, 5 up, 6 in 2024-05-26T11:56:48.719 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:48.719 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.719 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:48 smithi156 ceph-mon[32025]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: osdmap e35: 6 total, 5 up, 6 in 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: from='osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:48 smithi186 ceph-mon[36412]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:56:48.961 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 11:56:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T11:56:48.803+0000 7f7bbf9c4700 -1 osd.5 0 waiting for initial osdmap 2024-05-26T11:56:48.962 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 11:56:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T11:56:48.811+0000 7f7bb77af700 -1 osd.5 36 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:56:49.643 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:49 smithi192 ceph-mon[36545]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:56:49.643 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:49 smithi192 ceph-mon[36545]: osdmap e36: 6 total, 5 up, 6 in 2024-05-26T11:56:49.644 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:49 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.644 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:49 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.644 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:49 smithi192 ceph-mon[36545]: from='osd.5 ' entity='osd.5' 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.00835007 s, 1.3 GB/s 2024-05-26T11:56:49.702 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-05-26T11:56:49.702 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:49 smithi156 ceph-mon[32025]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:56:49.703 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:49 smithi156 ceph-mon[32025]: osdmap e36: 6 total, 5 up, 6 in 2024-05-26T11:56:49.703 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:49 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.703 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:49 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.703 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:49 smithi156 ceph-mon[32025]: from='osd.5 ' entity='osd.5' 2024-05-26T11:56:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:49 smithi186 ceph-mon[36412]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:56:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:49 smithi186 ceph-mon[36412]: osdmap e36: 6 total, 5 up, 6 in 2024-05-26T11:56:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:49 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:49 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:49 smithi186 ceph-mon[36412]: from='osd.5 ' entity='osd.5' 2024-05-26T11:56:50.422 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_2 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] boot 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: osdmap e37: 6 total, 6 up, 6 in 2024-05-26T11:56:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:50.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:56:50.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:56:50.647 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] boot 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: osdmap e37: 6 total, 6 up, 6 in 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:50.648 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.674 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:56:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:56:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:56:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-05-26T11:56:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: osd.5 [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] boot 2024-05-26T11:56:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: osdmap e37: 6 total, 6 up, 6 in 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:56:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:56:51.622 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:51 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi192 2024-05-26T11:56:51.622 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:51 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi192 to 8544M 2024-05-26T11:56:51.622 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:51 smithi156 ceph-mon[32025]: osdmap e38: 6 total, 6 up, 6 in 2024-05-26T11:56:51.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:51 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi192 2024-05-26T11:56:51.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:51 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi192 to 8544M 2024-05-26T11:56:51.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:51 smithi186 ceph-mon[36412]: osdmap e38: 6 total, 6 up, 6 in 2024-05-26T11:56:51.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:51 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi192 2024-05-26T11:56:51.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:51 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi192 to 8544M 2024-05-26T11:56:51.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:51 smithi192 ceph-mon[36545]: osdmap e38: 6 total, 6 up, 6 in 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 -- 172.21.15.156:0/1814939051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 msgr2=0x7f9944103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 --2- 172.21.15.156:0/1814939051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944103b00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f99340099c0 tx=0x7f9934009cd0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 -- 172.21.15.156:0/1814939051 shutdown_connections 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 --2- 172.21.15.156:0/1814939051 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9944104040 0x7f994410a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 --2- 172.21.15.156:0/1814939051 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99441024a0 0x7f99441028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 --2- 172.21.15.156:0/1814939051 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944103b00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 -- 172.21.15.156:0/1814939051 >> 172.21.15.156:0/1814939051 conn(0x7f99440fda40 msgr2=0x7f99440ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 -- 172.21.15.156:0/1814939051 shutdown_connections 2024-05-26T11:56:52.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.372+0000 7f994c631700 1 -- 172.21.15.156:0/1814939051 wait complete. 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 Processor -- start 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 -- start start 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99441024a0 0x7f9944106430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 0x7f9944104fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f994410df90 con 0x7f99441036a0 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f994410de10 con 0x7f9944104040 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994c631700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f994410e110 con 0x7f99441024a0 2024-05-26T11:56:52.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:45614/0 (socket says 172.21.15.156:45614) 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 -- 172.21.15.156:0/3997693089 learned_addr learned my addr 172.21.15.156:0/3997693089 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994a3cd700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99441024a0 0x7f9944106430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994abce700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 0x7f9944104fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 -- 172.21.15.156:0/3997693089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99441024a0 msgr2=0x7f9944106430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99441024a0 0x7f9944106430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 -- 172.21.15.156:0/3997693089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 msgr2=0x7f9944104fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 0x7f9944104fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f9949bcc700 1 -- 172.21.15.156:0/3997693089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9944105870 con 0x7f99441036a0 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.373+0000 7f994abce700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 0x7f9944104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f9949bcc700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f993400b580 tx=0x7f9934009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:52.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934027070 con 0x7f99441036a0 2024-05-26T11:56:52.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99340040a0 con 0x7f99441036a0 2024-05-26T11:56:52.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99441a38e0 con 0x7f99441036a0 2024-05-26T11:56:52.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934022d60 con 0x7f99441036a0 2024-05-26T11:56:52.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.374+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99441a3dc0 con 0x7f99441036a0 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.375+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994406a230 con 0x7f99441036a0 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.376+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f993402a070 con 0x7f99441036a0 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.377+0000 7f993b7fe700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 0x7f99300b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.377+0000 7f994a3cd700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 0x7f99300b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.377+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 4533+0+0 (secure 0 0 0) 0x7f99340e2310 con 0x7f99441036a0 2024-05-26T11:56:52.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.377+0000 7f994a3cd700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 0x7f99300b4dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9940009550 tx=0x7f99400091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:56:52.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.380+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9934013070 con 0x7f99441036a0 2024-05-26T11:56:52.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:52 smithi156 ceph-mon[32025]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:52.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:52 smithi156 ceph-mon[32025]: osdmap e39: 6 total, 6 up, 6 in 2024-05-26T11:56:52.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:56:52.597+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f99441a40e0 con 0x7f99300b2900 2024-05-26T11:56:52.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:52 smithi186 ceph-mon[36412]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:52.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:52 smithi186 ceph-mon[36412]: osdmap e39: 6 total, 6 up, 6 in 2024-05-26T11:56:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:52 smithi192 ceph-mon[36545]: pgmap v140: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:52 smithi192 ceph-mon[36545]: osdmap e39: 6 total, 6 up, 6 in 2024-05-26T11:56:53.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:53 smithi156 ceph-mon[32025]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:53.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:53.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:53.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:53.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:53 smithi186 ceph-mon[36412]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:53.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:53.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:53.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:53.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:53 smithi192 ceph-mon[36545]: from='client.14382 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:56:53.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:56:53.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:56:53.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:56:54.535 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:54 smithi156 ceph-mon[32025]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:54.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:54 smithi186 ceph-mon[36412]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:54.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:54 smithi192 ceph-mon[36545]: pgmap v142: 1 pgs: 1 peering; 577 KiB data, 193 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:56.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:56.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]: dispatch 2024-05-26T11:56:56.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]': finished 2024-05-26T11:56:56.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: osdmap e40: 7 total, 6 up, 7 in 2024-05-26T11:56:56.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:56:56.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:56 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3527792246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:56.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:56.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]: dispatch 2024-05-26T11:56:56.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]': finished 2024-05-26T11:56:56.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: osdmap e40: 7 total, 6 up, 7 in 2024-05-26T11:56:56.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:56:56.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:56 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3527792246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: pgmap v143: 1 pgs: 1 peering; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]: dispatch 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2400434043' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7f4d15c5-136b-4322-8bd9-4069355ccafb"}]': finished 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: osdmap e40: 7 total, 6 up, 7 in 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:56:56.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:56 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3527792246' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:56:58.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:56:58 smithi186 ceph-mon[36412]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-26T11:56:58.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:56:58 smithi156 ceph-mon[32025]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-26T11:56:58.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:56:58 smithi192 ceph-mon[36545]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-05-26T11:57:00.625 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:00 smithi156 ceph-mon[32025]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-26T11:57:00.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:00 smithi186 ceph-mon[36412]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-26T11:57:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:00 smithi192 ceph-mon[36545]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-26T11:57:02.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:02 smithi156 ceph-mon[32025]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-26T11:57:02.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:02 smithi186 ceph-mon[36412]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-26T11:57:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:02 smithi192 ceph-mon[36545]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-05-26T11:57:03.674 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-26T11:57:03.674 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:03.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-26T11:57:03.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:03.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-26T11:57:03.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:04.706 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:04 smithi156 ceph-mon[32025]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:04.707 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:04 smithi156 ceph-mon[32025]: Deploying daemon osd.6 on smithi156 2024-05-26T11:57:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:04 smithi186 ceph-mon[36412]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:04 smithi186 ceph-mon[36412]: Deploying daemon osd.6 on smithi156 2024-05-26T11:57:04.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:04 smithi192 ceph-mon[36545]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:04.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:04 smithi192 ceph-mon[36545]: Deploying daemon osd.6 on smithi156 2024-05-26T11:57:06.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:06 smithi186 ceph-mon[36412]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:06.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:06 smithi156 ceph-mon[32025]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:06 smithi192 ceph-mon[36545]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:57:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:08 smithi186 ceph-mon[36412]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:08.730 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:08 smithi156 ceph-mon[32025]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:08 smithi192 ceph-mon[36545]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:10.474 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:10 smithi156 ceph-mon[32025]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:10.475 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.475 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.475 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:10 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:10.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:10 smithi186 ceph-mon[36412]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:10.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:10 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:10 smithi192 ceph-mon[36545]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:10 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:12 smithi156 ceph-mon[32025]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:12.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:12 smithi186 ceph-mon[36412]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:12.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:12 smithi192 ceph-mon[36545]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:13.216 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 6 on host 'smithi156' 2024-05-26T11:57:13.217 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.215+0000 7f993b7fe700 1 -- 172.21.15.156:0/3997693089 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f99441a40e0 con 0x7f99300b2900 2024-05-26T11:57:13.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 msgr2=0x7f99300b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:13.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 0x7f99300b4dc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9940009550 tx=0x7f99400091b0 comp rx=0 tx=0).stop 2024-05-26T11:57:13.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 msgr2=0x7f9944104a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:13.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f993400b580 tx=0x7f9934009580 comp rx=0 tx=0).stop 2024-05-26T11:57:13.219 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 shutdown_connections 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f99300b2900 0x7f99300b4dc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9944104040 0x7f9944104fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f99441024a0 0x7f9944106430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 --2- 172.21.15.156:0/3997693089 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f99441036a0 0x7f9944104a80 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.218+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 >> 172.21.15.156:0/3997693089 conn(0x7f99440fda40 msgr2=0x7f994406b290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.219+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 shutdown_connections 2024-05-26T11:57:13.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:57:13.219+0000 7f994c631700 1 -- 172.21.15.156:0/3997693089 wait complete. 2024-05-26T11:57:13.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:13 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-26T11:57:13.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:13 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:13.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:13.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:13 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:13.827 DEBUG:teuthology.orchestra.run.smithi156:osd.6> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.6.service 2024-05-26T11:57:13.830 INFO:tasks.cephadm:Deploying osd.7 on smithi186 with /dev/vg_nvme/lv_2... 2024-05-26T11:57:13.830 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-26T11:57:14.077 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:57:14.450 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:14 smithi186 ceph-mon[36412]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:14.450 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:14 smithi186 ceph-mon[36412]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-26T11:57:14.450 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:14 smithi186 ceph-mon[36412]: osdmap e41: 7 total, 6 up, 7 in 2024-05-26T11:57:14.450 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:14 smithi186 ceph-mon[36412]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:57:14.451 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:14 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:14.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:14 smithi156 ceph-mon[32025]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:14.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:14 smithi156 ceph-mon[32025]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-26T11:57:14.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:14 smithi156 ceph-mon[32025]: osdmap e41: 7 total, 6 up, 7 in 2024-05-26T11:57:14.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:14 smithi156 ceph-mon[32025]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:57:14.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:14 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:14.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 11:57:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T11:57:14.260+0000 7f3495520700 -1 osd.6 0 waiting for initial osdmap 2024-05-26T11:57:14.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 11:57:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T11:57:14.270+0000 7f3491313700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:57:14.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:14 smithi192 ceph-mon[36545]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:14.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:14 smithi192 ceph-mon[36545]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-26T11:57:14.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:14 smithi192 ceph-mon[36545]: osdmap e41: 7 total, 6 up, 7 in 2024-05-26T11:57:14.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:14 smithi192 ceph-mon[36545]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:57:14.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:14 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.495 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:15 smithi186 ceph-mon[36412]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:57:15.495 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:15 smithi186 ceph-mon[36412]: osdmap e42: 7 total, 6 up, 7 in 2024-05-26T11:57:15.495 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.495 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:15 smithi156 ceph-mon[32025]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:57:15.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:15 smithi156 ceph-mon[32025]: osdmap e42: 7 total, 6 up, 7 in 2024-05-26T11:57:15.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:15 smithi192 ceph-mon[36545]: from='osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:57:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:15 smithi192 ceph-mon[36545]: osdmap e42: 7 total, 6 up, 7 in 2024-05-26T11:57:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:15.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114141 s, 919 MB/s 2024-05-26T11:57:16.219 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-26T11:57:16.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:57:16.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:57:16.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:16.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] boot 2024-05-26T11:57:16.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: osdmap e43: 7 total, 7 up, 7 in 2024-05-26T11:57:16.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:16 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:57:16.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:57:16.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:16.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] boot 2024-05-26T11:57:16.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: osdmap e43: 7 total, 7 up, 7 in 2024-05-26T11:57:16.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:16 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: osd.6 [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] boot 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: osdmap e43: 7 total, 7 up, 7 in 2024-05-26T11:57:16.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:16 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-26T11:57:16.890 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_2 2024-05-26T11:57:17.136 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: osdmap e44: 7 total, 7 up, 7 in 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:17.516 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: osdmap e44: 7 total, 7 up, 7 in 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:17.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: osdmap e44: 7 total, 7 up, 7 in 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:17.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:17.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:17.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:17.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:18 smithi156 ceph-mon[32025]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:18.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:18 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi156 2024-05-26T11:57:18.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:18 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi156 to 5696M 2024-05-26T11:57:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:18 smithi192 ceph-mon[36545]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:18 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi156 2024-05-26T11:57:18.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:18 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi156 to 5696M 2024-05-26T11:57:18.639 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:18 smithi186 ceph-mon[36412]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:18.639 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:18 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi156 2024-05-26T11:57:18.639 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:18 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi156 to 5696M 2024-05-26T11:57:18.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 -- 172.21.15.186:0/3513649042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 msgr2=0x7f23a8102c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:18.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 --2- 172.21.15.186:0/3513649042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a8102c80 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2398009990 tx=0x7f2398009ca0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.881 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 -- 172.21.15.186:0/3513649042 shutdown_connections 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 --2- 172.21.15.186:0/3513649042 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a8103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 --2- 172.21.15.186:0/3513649042 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 0x7f23a81022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 --2- 172.21.15.186:0/3513649042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a8102c80 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 -- 172.21.15.186:0/3513649042 >> 172.21.15.186:0/3513649042 conn(0x7f23a80fda40 msgr2=0x7f23a80ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 -- 172.21.15.186:0/3513649042 shutdown_connections 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 -- 172.21.15.186:0/3513649042 wait complete. 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.877+0000 7f23aed51700 1 Processor -- start 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 -- start start 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 0x7f23a819b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a81a2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:18.882 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f23a8111200 con 0x7f23a81031c0 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f23a8111080 con 0x7f23a8102800 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f23a8111380 con 0x7f23a8101ea0 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46030/0 (socket says 172.21.15.186:46030) 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23acaed700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 0x7f23a819b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 -- 172.21.15.186:0/416319502 learned_addr learned my addr 172.21.15.186:0/416319502 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23ad2ee700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a81a2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 -- 172.21.15.186:0/416319502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 msgr2=0x7f23a819b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 0x7f23a819b940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 -- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 msgr2=0x7f23a81a2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a81a2f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 -- 172.21.15.186:0/416319502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a81a35a0 con 0x7f23a8102800 2024-05-26T11:57:18.883 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23ad2ee700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a81a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239ffff700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2398006010 tx=0x7f23980047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2398027070 con 0x7f23a8102800 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23a81a3830 con 0x7f23a8102800 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2398004080 con 0x7f23a8102800 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.878+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2398022d70 con 0x7f23a8102800 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.879+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23a81a3c80 con 0x7f23a8102800 2024-05-26T11:57:18.884 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.880+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23a806a230 con 0x7f23a8102800 2024-05-26T11:57:18.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.882+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f2398004230 con 0x7f23a8102800 2024-05-26T11:57:18.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.882+0000 7f239dffb700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 0x7f23900b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:18.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.882+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4965+0+0 (secure 0 0 0) 0x7f23980e2490 con 0x7f23a8102800 2024-05-26T11:57:18.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.883+0000 7f23acaed700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 0x7f23900b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:18.885 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.883+0000 7f23acaed700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 0x7f23900b4d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009550 tx=0x7f23a40091b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:57:18.886 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:18.886+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2398030080 con 0x7f23a8102800 2024-05-26T11:57:19.095 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:19.095+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f23a8063980 con 0x7f23900b2880 2024-05-26T11:57:19.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:19 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:19.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:19 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:19.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:19 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:19.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:19 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:19.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:19 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:19.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:19 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:19.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:19 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:19.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:19 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:19.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:19 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:20.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:20 smithi156 ceph-mon[32025]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:20.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:20 smithi156 ceph-mon[32025]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:20.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:20 smithi192 ceph-mon[36545]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:20.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:20 smithi192 ceph-mon[36545]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:20 smithi186 ceph-mon[36412]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:20.714 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:20 smithi186 ceph-mon[36412]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/1600729108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]': finished 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: osdmap e45: 8 total, 7 up, 8 in 2024-05-26T11:57:22.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/1600729108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]': finished 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: osdmap e45: 8 total, 7 up, 8 in 2024-05-26T11:57:22.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/1600729108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]: dispatch 2024-05-26T11:57:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fe4f82a9-bb86-4323-b95f-69429a432679"}]': finished 2024-05-26T11:57:22.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: osdmap e45: 8 total, 7 up, 8 in 2024-05-26T11:57:22.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:23.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:23 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/939494202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:23.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:23 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/939494202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:23.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:23 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/939494202' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:24.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:24 smithi192 ceph-mon[36545]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:24 smithi186 ceph-mon[36412]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:24.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:24 smithi156 ceph-mon[32025]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:26.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:26 smithi192 ceph-mon[36545]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:26.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:26 smithi186 ceph-mon[36412]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:26.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:26 smithi156 ceph-mon[32025]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:28 smithi186 ceph-mon[36412]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:28.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:28 smithi192 ceph-mon[36545]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:28.596 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:28 smithi156 ceph-mon[32025]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:29.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-26T11:57:29.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:29.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-26T11:57:29.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:29.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-26T11:57:29.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:30.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:30 smithi192 ceph-mon[36545]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:30.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:30 smithi192 ceph-mon[36545]: Deploying daemon osd.7 on smithi186 2024-05-26T11:57:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:30 smithi186 ceph-mon[36412]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:30 smithi186 ceph-mon[36412]: Deploying daemon osd.7 on smithi186 2024-05-26T11:57:30.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:30 smithi156 ceph-mon[32025]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:30.821 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:30 smithi156 ceph-mon[32025]: Deploying daemon osd.7 on smithi186 2024-05-26T11:57:32.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:32 smithi192 ceph-mon[36545]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:32.691 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:32 smithi186 ceph-mon[36412]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:32.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:32 smithi156 ceph-mon[32025]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:34.472 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:34 smithi186 ceph-mon[36412]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:34 smithi192 ceph-mon[36545]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:34 smithi156 ceph-mon[32025]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:36.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:36 smithi156 ceph-mon[32025]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:36.474 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.475 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.475 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:36.475 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:36 smithi192 ceph-mon[36545]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:36.475 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:36 smithi186 ceph-mon[36412]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:38.180 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:38 smithi186 ceph-mon[36412]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:38.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:38 smithi192 ceph-mon[36545]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:38 smithi156 ceph-mon[32025]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:38.560 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 7 on host 'smithi186' 2024-05-26T11:57:38.561 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.560+0000 7f239dffb700 1 -- 172.21.15.186:0/416319502 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f23a8063980 con 0x7f23900b2880 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.562+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 msgr2=0x7f23900b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.562+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 0x7f23900b4d40 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f23a4009550 tx=0x7f23a40091b0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.562+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 msgr2=0x7f23a819be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.562+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2398006010 tx=0x7f23980047b0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 shutdown_connections 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f23900b2880 0x7f23900b4d40 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.563 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f23a81031c0 0x7f23a81a2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f23a8101ea0 0x7f23a819b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 --2- 172.21.15.186:0/416319502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23a8102800 0x7f23a819be80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:38.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 >> 172.21.15.186:0/416319502 conn(0x7f23a80fda40 msgr2=0x7f23a80ff540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:57:38.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 shutdown_connections 2024-05-26T11:57:38.564 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:57:38.563+0000 7f23aed51700 1 -- 172.21.15.186:0/416319502 wait complete. 2024-05-26T11:57:39.361 DEBUG:teuthology.orchestra.run.smithi186:osd.7> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.7.service 2024-05-26T11:57:39.364 INFO:tasks.cephadm:Deploying osd.8 on smithi192 with /dev/vg_nvme/lv_2... 2024-05-26T11:57:39.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-26T11:57:39.612 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:57:39.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:39 smithi186 ceph-mon[36412]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:39.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:39 smithi186 ceph-mon[36412]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:39.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.733 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.734 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:39 smithi192 ceph-mon[36545]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:39.734 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:39 smithi192 ceph-mon[36545]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:39.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:39.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:39 smithi156 ceph-mon[32025]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:39.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:39 smithi156 ceph-mon[32025]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-26T11:57:40.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:40.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: osdmap e46: 8 total, 7 up, 8 in 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:40 smithi186 ceph-mon[36412]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:40.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 11:57:40 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T11:57:40.521+0000 7f173643a700 -1 osd.7 0 waiting for initial osdmap 2024-05-26T11:57:40.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 11:57:40 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T11:57:40.529+0000 7f173222d700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:57:40.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: osdmap e46: 8 total, 7 up, 8 in 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:40 smithi156 ceph-mon[32025]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: osdmap e46: 8 total, 7 up, 8 in 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:40 smithi192 ceph-mon[36545]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:57:41.654 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0114048 s, 919 MB/s 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 2024-05-26T11:57:41.655 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-05-26T11:57:41.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:41 smithi186 ceph-mon[36412]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:57:41.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:41 smithi186 ceph-mon[36412]: osdmap e47: 8 total, 7 up, 8 in 2024-05-26T11:57:41.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:41.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:41 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:41.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:41 smithi156 ceph-mon[32025]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:57:41.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:41 smithi156 ceph-mon[32025]: osdmap e47: 8 total, 7 up, 8 in 2024-05-26T11:57:41.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:41.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:41 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:41.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:41 smithi192 ceph-mon[36545]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:57:41.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:41 smithi192 ceph-mon[36545]: osdmap e47: 8 total, 7 up, 8 in 2024-05-26T11:57:41.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:41.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:41 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:42.401 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_2 2024-05-26T11:57:42.636 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:57:42.757 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] boot 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: osdmap e48: 8 total, 8 up, 8 in 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:42.758 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:42 smithi192 ceph-mon[36545]: osdmap e49: 8 total, 8 up, 8 in 2024-05-26T11:57:42.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:57:42.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:57:42.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:42.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] boot 2024-05-26T11:57:42.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: osdmap e48: 8 total, 8 up, 8 in 2024-05-26T11:57:42.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:42.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:42 smithi156 ceph-mon[32025]: osdmap e49: 8 total, 8 up, 8 in 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 200 MiB used, 626 GiB / 626 GiB avail 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: osd.7 [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] boot 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: osdmap e48: 8 total, 8 up, 8 in 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-26T11:57:42.950 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:42 smithi186 ceph-mon[36412]: osdmap e49: 8 total, 8 up, 8 in 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: pgmap v177: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi186 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:43 smithi192 ceph-mon[36545]: osdmap e50: 8 total, 8 up, 8 in 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: pgmap v177: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi186 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.296 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.297 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.297 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:43 smithi156 ceph-mon[32025]: osdmap e50: 8 total, 8 up, 8 in 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 -- 172.21.15.192:0/1852905538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 msgr2=0x7f965c103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 --2- 172.21.15.192:0/1852905538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c103af0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f964c009990 tx=0x7f964c009ca0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 -- 172.21.15.192:0/1852905538 shutdown_connections 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 --2- 172.21.15.192:0/1852905538 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c104030 0x7f965c10a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 --2- 172.21.15.192:0/1852905538 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c102490 0x7f965c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 --2- 172.21.15.192:0/1852905538 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c103af0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 -- 172.21.15.192:0/1852905538 >> 172.21.15.192:0/1852905538 conn(0x7f965c0fda70 msgr2=0x7f965c0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 -- 172.21.15.192:0/1852905538 shutdown_connections 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.316+0000 7f9664699700 1 -- 172.21.15.192:0/1852905538 wait complete. 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 Processor -- start 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 -- start start 2024-05-26T11:57:44.320 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c102490 0x7f965c06c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c104030 0x7f965c1a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10df30 con 0x7f965c102490 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10ddb0 con 0x7f965c104030 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9664699700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f965c10e0b0 con 0x7f965c103690 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9661c34700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9661c34700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:43530/0 (socket says 172.21.15.192:43530) 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9661c34700 1 -- 172.21.15.192:0/3706111915 learned_addr learned my addr 172.21.15.192:0/3706111915 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9662435700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c102490 0x7f965c06c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.317+0000 7f9662c36700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c104030 0x7f965c1a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 -- 172.21.15.192:0/3706111915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c104030 msgr2=0x7f965c1a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:44.321 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c104030 0x7f965c1a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 -- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c102490 msgr2=0x7f965c06c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c102490 0x7f965c06c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 -- 172.21.15.192:0/3706111915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f965c1a12d0 con 0x7f965c103690 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9661c34700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f964c0036f0 tx=0x7f964c009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c027070 con 0x7f965c103690 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f964c00bd90 con 0x7f965c103690 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f964c022d40 con 0x7f965c103690 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f965c1a1560 con 0x7f965c103690 2024-05-26T11:57:44.322 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.318+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f965c1a19b0 con 0x7f965c103690 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.319+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f965c02e560 con 0x7f965c103690 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.320+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f964c02a070 con 0x7f965c103690 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.321+0000 7f96537fe700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 0x7f96480b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.321+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f964c0e2db0 con 0x7f965c103690 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.321+0000 7f9662435700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 0x7f96480b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:57:44.323 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.322+0000 7f9662435700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 0x7f96480b4d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f965c1034f0 tx=0x7f965800b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:57:44.326 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.325+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f964c031020 con 0x7f965c103690 2024-05-26T11:57:44.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: pgmap v177: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:44.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi186 2024-05-26T11:57:44.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi186 to 5696M 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:57:44.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:43 smithi186 ceph-mon[36412]: osdmap e50: 8 total, 8 up, 8 in 2024-05-26T11:57:44.535 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:57:44.533+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f965c063980 con 0x7f96480b2880 2024-05-26T11:57:45.249 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:44 smithi192 ceph-mon[36545]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:45.249 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:44 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:45.249 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:44 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:45.250 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:44 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:45.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:44 smithi156 ceph-mon[32025]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:45.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:45.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:45.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:44 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:45.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:44 smithi186 ceph-mon[36412]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:57:45.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:44 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:57:45.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:44 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:57:45.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:44 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:46.252 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:45 smithi192 ceph-mon[36545]: pgmap v179: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:46.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:45 smithi156 ceph-mon[32025]: pgmap v179: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:46.377 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:45 smithi186 ceph-mon[36412]: pgmap v179: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: pgmap v180: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/4046048538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]': finished 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: osdmap e51: 9 total, 8 up, 9 in 2024-05-26T11:57:48.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:47 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: pgmap v180: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/4046048538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]': finished 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: osdmap e51: 9 total, 8 up, 9 in 2024-05-26T11:57:48.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:47 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: pgmap v180: 1 pgs: 1 peering; 0 B data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/4046048538' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]: dispatch 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b"}]': finished 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: osdmap e51: 9 total, 8 up, 9 in 2024-05-26T11:57:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:47 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:57:49.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:49 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/3503117786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:49.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:49 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/3503117786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:49.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:49 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/3503117786' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:57:50.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:50 smithi156 ceph-mon[32025]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:50.321 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:50 smithi192 ceph-mon[36545]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:50.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:50 smithi186 ceph-mon[36412]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:52.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:52 smithi156 ceph-mon[32025]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:52.321 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:52 smithi192 ceph-mon[36545]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:52.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:52 smithi186 ceph-mon[36412]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:54.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:54 smithi192 ceph-mon[36545]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:54.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:54 smithi156 ceph-mon[32025]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:54.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:54 smithi186 ceph-mon[36412]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:55.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-26T11:57:55.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:55.321 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-26T11:57:55.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:55.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-26T11:57:55.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:57:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:56 smithi192 ceph-mon[36545]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:56.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:56 smithi192 ceph-mon[36545]: Deploying daemon osd.8 on smithi192 2024-05-26T11:57:56.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:56 smithi156 ceph-mon[32025]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:56.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:56 smithi156 ceph-mon[32025]: Deploying daemon osd.8 on smithi192 2024-05-26T11:57:56.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:56 smithi186 ceph-mon[36412]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:56.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:56 smithi186 ceph-mon[36412]: Deploying daemon osd.8 on smithi192 2024-05-26T11:57:58.191 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:57:58 smithi192 ceph-mon[36545]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:58.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:57:58 smithi156 ceph-mon[32025]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:57:58.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:57:58 smithi186 ceph-mon[36412]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:00.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:00 smithi156 ceph-mon[32025]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:00.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:00 smithi192 ceph-mon[36545]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:00.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:00 smithi186 ceph-mon[36412]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:01.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:01.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:01.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:01 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:01.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:01 smithi186 ceph-mon[36412]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:02.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:02.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:02.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:01 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:02.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:01 smithi156 ceph-mon[32025]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:01 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:02.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:01 smithi192 ceph-mon[36545]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:04.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:03 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:04.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:04.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:04.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:03 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:04.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:04.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:04.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:03 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:04.753 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.753+0000 7f96537fe700 1 -- 172.21.15.192:0/3706111915 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f965c063980 con 0x7f96480b2880 2024-05-26T11:58:04.754 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 8 on host 'smithi192' 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 msgr2=0x7f96480b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 0x7f96480b4d40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f965c1034f0 tx=0x7f965800b410 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 msgr2=0x7f965c19e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f964c0036f0 tx=0x7f964c009580 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 shutdown_connections 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f96480b2880 0x7f96480b4d40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f965c104030 0x7f965c1a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f965c102490 0x7f965c06c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 --2- 172.21.15.192:0/3706111915 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f965c103690 0x7f965c19e3b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:04.756 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 >> 172.21.15.192:0/3706111915 conn(0x7f965c0fda70 msgr2=0x7f965c106a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:58:04.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 shutdown_connections 2024-05-26T11:58:04.757 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:58:04.755+0000 7f9664699700 1 -- 172.21.15.192:0/3706111915 wait complete. 2024-05-26T11:58:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:04 smithi192 ceph-mon[36545]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:04 smithi192 ceph-mon[36545]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:05.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:04 smithi186 ceph-mon[36412]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:04 smithi186 ceph-mon[36412]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:05.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:04 smithi156 ceph-mon[32025]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:04 smithi156 ceph-mon[32025]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-26T11:58:05.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:05.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:05.425 DEBUG:teuthology.orchestra.run.smithi192:osd.8> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.8.service 2024-05-26T11:58:05.427 INFO:tasks.cephadm:Deploying osd.9 on smithi156 with /dev/vg_nvme/lv_1... 2024-05-26T11:58:05.427 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-26T11:58:05.677 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:58:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:05.907 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-26T11:58:05.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: osdmap e52: 9 total, 8 up, 9 in 2024-05-26T11:58:05.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:05.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:05.908 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:05 smithi192 ceph-mon[36545]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:05.908 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 11:58:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T11:58:05.839+0000 7fdee0edb700 -1 osd.8 0 waiting for initial osdmap 2024-05-26T11:58:05.908 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 11:58:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T11:58:05.847+0000 7fdedccce700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: osdmap e52: 9 total, 8 up, 9 in 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:06.157 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:05 smithi156 ceph-mon[32025]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:06.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:06.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-26T11:58:06.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: osdmap e52: 9 total, 8 up, 9 in 2024-05-26T11:58:06.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:06.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: from='osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:06.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:05 smithi186 ceph-mon[36412]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:58:06.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:06 smithi192 ceph-mon[36545]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:58:06.983 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:06 smithi192 ceph-mon[36545]: osdmap e53: 9 total, 8 up, 9 in 2024-05-26T11:58:06.983 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:06.983 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:06.983 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.109 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:06 smithi156 ceph-mon[32025]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:58:07.109 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:06 smithi156 ceph-mon[32025]: osdmap e53: 9 total, 8 up, 9 in 2024-05-26T11:58:07.109 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.109 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.109 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:06 smithi186 ceph-mon[36412]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:58:07.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:06 smithi186 ceph-mon[36412]: osdmap e53: 9 total, 8 up, 9 in 2024-05-26T11:58:07.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:07.851 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-26T11:58:07.851 INFO:teuthology.orchestra.run.smithi156.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-26T11:58:07.851 INFO:teuthology.orchestra.run.smithi156.stderr: stderr: 10+0 records in 2024-05-26T11:58:07.851 INFO:teuthology.orchestra.run.smithi156.stderr:10+0 records out 2024-05-26T11:58:07.851 INFO:teuthology.orchestra.run.smithi156.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.00849381 s, 1.2 GB/s 2024-05-26T11:58:07.852 INFO:teuthology.orchestra.run.smithi156.stderr:--> Zapping successful for: 2024-05-26T11:58:08.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:58:08.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:58:08.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:08.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] boot 2024-05-26T11:58:08.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: osdmap e54: 9 total, 9 up, 9 in 2024-05-26T11:58:08.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:08.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:07 smithi192 ceph-mon[36545]: osdmap e55: 9 total, 9 up, 9 in 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] boot 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: osdmap e54: 9 total, 9 up, 9 in 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:08.180 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:07 smithi156 ceph-mon[32025]: osdmap e55: 9 total, 9 up, 9 in 2024-05-26T11:58:08.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: osd.8 [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] boot 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: osdmap e54: 9 total, 9 up, 9 in 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-26T11:58:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:07 smithi186 ceph-mon[36412]: osdmap e55: 9 total, 9 up, 9 in 2024-05-26T11:58:08.625 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi156:vg_nvme/lv_1 2024-05-26T11:58:08.872 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:58:09.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: osdmap e56: 9 total, 9 up, 9 in 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi192 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi192 to 5696M 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:09.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:09 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: osdmap e56: 9 total, 9 up, 9 in 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi192 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi192 to 5696M 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:09.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:09 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: osdmap e56: 9 total, 9 up, 9 in 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi192 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi192 to 5696M 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:09.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:09 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:10.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 -- 172.21.15.156:0/3540705710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 msgr2=0x7f9db0108d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:10.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 --2- 172.21.15.156:0/3540705710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0108d50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9da0009990 tx=0x7f9da0009ca0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 -- 172.21.15.156:0/3540705710 shutdown_connections 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 --2- 172.21.15.156:0/3540705710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9db0109290 0x7f9db010b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 --2- 172.21.15.156:0/3540705710 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0104030 0x7f9db0106420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 --2- 172.21.15.156:0/3540705710 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0108d50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 -- 172.21.15.156:0/3540705710 >> 172.21.15.156:0/3540705710 conn(0x7f9db00fda40 msgr2=0x7f9db00ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 -- 172.21.15.156:0/3540705710 shutdown_connections 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.662+0000 7f9db76d5700 1 -- 172.21.15.156:0/3540705710 wait complete. 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 Processor -- start 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 -- start start 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9db0104030 0x7f9db0199750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:10.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 0x7f9db01a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9db010f060 con 0x7f9db0106960 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9db010eee0 con 0x7f9db0104030 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db76d5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9db010f1e0 con 0x7f9db0109290 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:33536/0 (socket says 172.21.15.156:33536) 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 -- 172.21.15.156:0/528905172 learned_addr learned my addr 172.21.15.156:0/528905172 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db5c72700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 0x7f9db01a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 -- 172.21.15.156:0/528905172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 msgr2=0x7f9db01a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 0x7f9db01a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 -- 172.21.15.156:0/528905172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9db0104030 msgr2=0x7f9db0199750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9db0104030 0x7f9db0199750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:10.666 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 -- 172.21.15.156:0/528905172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9db01a13b0 con 0x7f9db0106960 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db5c72700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 0x7f9db01a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.663+0000 7f9db4c70700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9da00059d0 tx=0x7f9da0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.664+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0027070 con 0x7f9db0106960 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.664+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9db01a1640 con 0x7f9db0106960 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.664+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9da0004080 con 0x7f9db0106960 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.664+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9da0022cf0 con 0x7f9db0106960 2024-05-26T11:58:10.667 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.664+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9db01a1af0 con 0x7f9db0106960 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.665+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9db006a230 con 0x7f9db0106960 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.666+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f9da002a070 con 0x7f9db0106960 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.667+0000 7f9da67fc700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 0x7f9d9c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.667+0000 7f9db5471700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 0x7f9d9c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.667+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7f9da00e2f10 con 0x7f9db0106960 2024-05-26T11:58:10.668 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.667+0000 7f9db5471700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 0x7f9d9c0b4d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9dac005950 tx=0x7f9dac0058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:58:10.671 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.670+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9da0031070 con 0x7f9db0106960 2024-05-26T11:58:10.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:10.884+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9db0063a40 con 0x7f9d9c0b2880 2024-05-26T11:58:10.885 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:10 smithi156 ceph-mon[32025]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:10.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:10 smithi186 ceph-mon[36412]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:11.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:10 smithi192 ceph-mon[36545]: pgmap v197: 1 pgs: 1 peering; 577 KiB data, 254 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:11.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:11.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:11.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:11.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:11.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:11.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:12.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:12.783 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:12 smithi156 ceph-mon[32025]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:12.784 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:12 smithi156 ceph-mon[32025]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:12.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:12 smithi186 ceph-mon[36412]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:12.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:12 smithi186 ceph-mon[36412]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:13.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:12 smithi192 ceph-mon[36545]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:13.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:12 smithi192 ceph-mon[36545]: from='client.14433 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi156:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:14.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-26T11:58:14.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]: dispatch 2024-05-26T11:58:14.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]': finished 2024-05-26T11:58:14.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: osdmap e57: 10 total, 9 up, 10 in 2024-05-26T11:58:14.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:14.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:14 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2195139115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:15.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-26T11:58:15.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]: dispatch 2024-05-26T11:58:15.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]': finished 2024-05-26T11:58:15.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: osdmap e57: 10 total, 9 up, 10 in 2024-05-26T11:58:15.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:15.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:14 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2195139115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]: dispatch 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2743099324' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ea261f2e-8e96-430c-8e46-c1280b252e50"}]': finished 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: osdmap e57: 10 total, 9 up, 10 in 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:14 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2195139115' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:16.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:16 smithi186 ceph-mon[36412]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-26T11:58:17.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:16 smithi156 ceph-mon[32025]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-26T11:58:17.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:16 smithi192 ceph-mon[36545]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-05-26T11:58:17.734 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:17 smithi156 ceph-mon[32025]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-26T11:58:17.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:17 smithi186 ceph-mon[36412]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-26T11:58:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:17 smithi192 ceph-mon[36545]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-05-26T11:58:20.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:19 smithi156 ceph-mon[32025]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:20.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:19 smithi192 ceph-mon[36545]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:20.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:19 smithi186 ceph-mon[36412]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:22.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:21 smithi192 ceph-mon[36545]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:22.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-26T11:58:22.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:22.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:21 smithi192 ceph-mon[36545]: Deploying daemon osd.9 on smithi156 2024-05-26T11:58:22.098 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:21 smithi156 ceph-mon[32025]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:22.098 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-26T11:58:22.098 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:22.098 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:21 smithi156 ceph-mon[32025]: Deploying daemon osd.9 on smithi156 2024-05-26T11:58:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:21 smithi186 ceph-mon[36412]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:58:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-26T11:58:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:22.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:21 smithi186 ceph-mon[36412]: Deploying daemon osd.9 on smithi156 2024-05-26T11:58:23.933 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:23 smithi156 ceph-mon[32025]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:24.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:23 smithi186 ceph-mon[36412]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:24.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:23 smithi192 ceph-mon[36545]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:25 smithi156 ceph-mon[32025]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:26.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:25 smithi186 ceph-mon[36412]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:25 smithi192 ceph-mon[36545]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:28.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:28 smithi156 ceph-mon[32025]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:28.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:28.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:28.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:28.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:28 smithi186 ceph-mon[36412]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:28.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:28.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:28.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:28.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:28 smithi192 ceph-mon[36545]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:28.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:28.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:28.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:30.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:30 smithi156 ceph-mon[32025]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:30 smithi186 ceph-mon[36412]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:30.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:30 smithi192 ceph-mon[36545]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:31.502 INFO:teuthology.orchestra.run.smithi156.stdout:Created osd(s) 9 on host 'smithi156' 2024-05-26T11:58:31.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.501+0000 7f9da67fc700 1 -- 172.21.15.156:0/528905172 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f9db0063a40 con 0x7f9d9c0b2880 2024-05-26T11:58:31.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 msgr2=0x7f9d9c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:31.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 0x7f9d9c0b4d40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9dac005950 tx=0x7f9dac0058e0 comp rx=0 tx=0).stop 2024-05-26T11:58:31.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 msgr2=0x7f9db0199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9da00059d0 tx=0x7f9da0009f90 comp rx=0 tx=0).stop 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 shutdown_connections 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f9d9c0b2880 0x7f9d9c0b4d40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9db0109290 0x7f9db01a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9db0104030 0x7f9db0199750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 --2- 172.21.15.156:0/528905172 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9db0106960 0x7f9db0199c90 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 >> 172.21.15.156:0/528905172 conn(0x7f9db00fda40 msgr2=0x7f9db0105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 shutdown_connections 2024-05-26T11:58:31.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:58:31.504+0000 7f9db76d5700 1 -- 172.21.15.156:0/528905172 wait complete. 2024-05-26T11:58:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:31 smithi156 ceph-mon[32025]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-26T11:58:31.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:31.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:31.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:31 smithi186 ceph-mon[36412]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:31 smithi192 ceph-mon[36545]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-26T11:58:32.227 DEBUG:teuthology.orchestra.run.smithi156:osd.9> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.9.service 2024-05-26T11:58:32.229 INFO:tasks.cephadm:Deploying osd.10 on smithi186 with /dev/vg_nvme/lv_1... 2024-05-26T11:58:32.229 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-26T11:58:32.477 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: osdmap e58: 10 total, 9 up, 10 in 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:32.599 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:32 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: osdmap e58: 10 total, 9 up, 10 in 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:32.676 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:32.677 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:32 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:32.677 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 11:58:32 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T11:58:32.462+0000 7f0289efb700 -1 osd.9 0 waiting for initial osdmap 2024-05-26T11:58:32.677 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 11:58:32 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T11:58:32.472+0000 7f0285cee700 -1 osd.9 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:58:32.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:32.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-26T11:58:32.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: osdmap e58: 10 total, 9 up, 10 in 2024-05-26T11:58:32.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]: dispatch 2024-05-26T11:58:32.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:32.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:32.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:32.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:32 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:33 smithi186 ceph-mon[36412]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:58:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:33 smithi186 ceph-mon[36412]: osdmap e59: 10 total, 9 up, 10 in 2024-05-26T11:58:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:33 smithi156 ceph-mon[32025]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:58:33.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:33 smithi156 ceph-mon[32025]: osdmap e59: 10 total, 9 up, 10 in 2024-05-26T11:58:33.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:33 smithi192 ceph-mon[36545]: from='osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi156", "root=default"]}]': finished 2024-05-26T11:58:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:33 smithi192 ceph-mon[36545]: osdmap e59: 10 total, 9 up, 10 in 2024-05-26T11:58:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10+0 records in 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr:10+0 records out 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114241 s, 918 MB/s 2024-05-26T11:58:34.608 INFO:teuthology.orchestra.run.smithi186.stderr:--> Zapping successful for: 2024-05-26T11:58:34.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:58:34.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:58:34.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:34.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] boot 2024-05-26T11:58:34.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: osdmap e60: 10 total, 10 up, 10 in 2024-05-26T11:58:34.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:34 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:58:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:58:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:34.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] boot 2024-05-26T11:58:34.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: osdmap e60: 10 total, 10 up, 10 in 2024-05-26T11:58:34.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:34 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:34.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:58:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:58:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-26T11:58:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: osd.9 [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] boot 2024-05-26T11:58:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: osdmap e60: 10 total, 10 up, 10 in 2024-05-26T11:58:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:34 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-26T11:58:35.309 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi186:vg_nvme/lv_1 2024-05-26T11:58:35.551 INFO:teuthology.orchestra.run.smithi186.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.b/config 2024-05-26T11:58:35.673 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:35 smithi186 ceph-mon[36412]: osdmap e61: 10 total, 10 up, 10 in 2024-05-26T11:58:35.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:35 smithi156 ceph-mon[32025]: osdmap e61: 10 total, 10 up, 10 in 2024-05-26T11:58:35.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:35 smithi192 ceph-mon[36545]: osdmap e61: 10 total, 10 up, 10 in 2024-05-26T11:58:36.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi156 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi156 to 4272M 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:36.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:36 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:36.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi156 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi156 to 4272M 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:36.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:36 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi156 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi156 to 4272M 2024-05-26T11:58:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:36.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:36.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:36.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:36 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.280+0000 7f6047d9d700 1 -- 172.21.15.186:0/3070910354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6040104040 msgr2=0x7f604010a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.280+0000 7f6047d9d700 1 --2- 172.21.15.186:0/3070910354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6040104040 0x7f604010a8f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f603c0099c0 tx=0x7f603c009cd0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 -- 172.21.15.186:0/3070910354 shutdown_connections 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 --2- 172.21.15.186:0/3070910354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6040104040 0x7f604010a8f0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 --2- 172.21.15.186:0/3070910354 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f60401024a0 0x7f60401028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 --2- 172.21.15.186:0/3070910354 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401036a0 0x7f6040103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 -- 172.21.15.186:0/3070910354 >> 172.21.15.186:0/3070910354 conn(0x7f60400fda60 msgr2=0x7f60400ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 -- 172.21.15.186:0/3070910354 shutdown_connections 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 -- 172.21.15.186:0/3070910354 wait complete. 2024-05-26T11:58:37.284 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 Processor -- start 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 -- start start 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 0x7f6040198560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.281+0000 7f6047d9d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 0x7f604019fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f604010df20 con 0x7f60401024a0 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f604010dda0 con 0x7f60401036a0 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f604010e0a0 con 0x7f6040104040 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:46598/0 (socket says 172.21.15.186:46598) 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 -- 172.21.15.186:0/1394786473 learned_addr learned my addr 172.21.15.186:0/1394786473 (peer_addr_for_me v2:172.21.15.186:0/0) 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f604633a700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 0x7f604019fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045b39700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 0x7f6040198560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:37.285 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 -- 172.21.15.186:0/1394786473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 msgr2=0x7f604019fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 0x7f604019fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 -- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 msgr2=0x7f6040198560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 0x7f6040198560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 -- 172.21.15.186:0/1394786473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60401a01c0 con 0x7f60401036a0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f604633a700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 0x7f604019fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045b39700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 0x7f6040198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6045338700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f603000d950 tx=0x7f603000dd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6030016070 con 0x7f60401036a0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6030010460 con 0x7f60401036a0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603000f600 con 0x7f60401036a0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60401a04b0 con 0x7f60401036a0 2024-05-26T11:58:37.286 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.282+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60401a0a40 con 0x7f60401036a0 2024-05-26T11:58:37.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.283+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f604006a230 con 0x7f60401036a0 2024-05-26T11:58:37.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.284+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f6030019070 con 0x7f60401036a0 2024-05-26T11:58:37.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.285+0000 7f6036ffd700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 0x7f602c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:58:37.287 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.285+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7f60300e15d0 con 0x7f60401036a0 2024-05-26T11:58:37.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.285+0000 7f6045b39700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 0x7f602c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:58:37.288 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.286+0000 7f6045b39700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 0x7f602c0b4dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6038009690 tx=0x7f6038009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:58:37.290 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.290+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f60300a9af0 con 0x7f60401036a0 2024-05-26T11:58:37.503 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:37.502+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f6040063980 con 0x7f602c0b2900 2024-05-26T11:58:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:38 smithi186 ceph-mon[36412]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:38 smithi186 ceph-mon[36412]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:38.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:38 smithi156 ceph-mon[32025]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:38.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:38 smithi156 ceph-mon[32025]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:38.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:38.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:38.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:38 smithi192 ceph-mon[36545]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:38 smithi192 ceph-mon[36545]: from='client.24362 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi186:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:58:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:58:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:58:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:40.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/133509014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]': finished 2024-05-26T11:58:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: osdmap e62: 11 total, 10 up, 11 in 2024-05-26T11:58:40.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/133509014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]': finished 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: osdmap e62: 11 total, 10 up, 11 in 2024-05-26T11:58:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/133509014' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]: dispatch 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "30ce9960-8b0d-423a-b02b-5896204ddbf1"}]': finished 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: osdmap e62: 11 total, 10 up, 11 in 2024-05-26T11:58:40.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:41.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:41 smithi156 ceph-mon[32025]: from='client.? 172.21.15.186:0/1674107967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:41.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:41 smithi192 ceph-mon[36545]: from='client.? 172.21.15.186:0/1674107967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:41 smithi186 ceph-mon[36412]: from='client.? 172.21.15.186:0/1674107967' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:58:42.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:42 smithi156 ceph-mon[32025]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:42.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:42 smithi192 ceph-mon[36545]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:42 smithi186 ceph-mon[36412]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:44.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:44 smithi186 ceph-mon[36412]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:44.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:44 smithi156 ceph-mon[32025]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:44.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:44 smithi192 ceph-mon[36545]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:46.706 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:46 smithi186 ceph-mon[36412]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:46.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:46 smithi156 ceph-mon[32025]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:46 smithi192 ceph-mon[36545]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:48.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:48 smithi186 ceph-mon[36412]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:48.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-26T11:58:48.667 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:48 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:48.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:48 smithi156 ceph-mon[32025]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:48.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-26T11:58:48.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:48 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:48 smithi192 ceph-mon[36545]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-26T11:58:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:48 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:49.776 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:49 smithi186 ceph-mon[36412]: Deploying daemon osd.10 on smithi186 2024-05-26T11:58:49.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:49 smithi156 ceph-mon[32025]: Deploying daemon osd.10 on smithi186 2024-05-26T11:58:49.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:49 smithi192 ceph-mon[36545]: Deploying daemon osd.10 on smithi186 2024-05-26T11:58:50.672 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:50 smithi186 ceph-mon[36412]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:50.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:50 smithi156 ceph-mon[32025]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:50.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:50 smithi192 ceph-mon[36545]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:52.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:52 smithi156 ceph-mon[32025]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:52 smithi192 ceph-mon[36545]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:52.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:52 smithi186 ceph-mon[36412]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:54 smithi186 ceph-mon[36412]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:54 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:54 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:54.695 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:54 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:54.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:54 smithi156 ceph-mon[32025]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:54.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:54 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:54.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:54 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:54.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:54 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:54.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:54 smithi192 ceph-mon[36545]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:54.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:54 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:54.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:54 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:54.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:54 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:56.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:56 smithi186 ceph-mon[36412]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:56.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:56 smithi156 ceph-mon[32025]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:56.821 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:56 smithi192 ceph-mon[36545]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:57.822 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.821+0000 7f6036ffd700 1 -- 172.21.15.186:0/1394786473 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f6040063980 con 0x7f602c0b2900 2024-05-26T11:58:57.823 INFO:teuthology.orchestra.run.smithi186.stdout:Created osd(s) 10 on host 'smithi186' 2024-05-26T11:58:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 msgr2=0x7f602c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 0x7f602c0b4dc0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6038009690 tx=0x7f6038009040 comp rx=0 tx=0).stop 2024-05-26T11:58:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 msgr2=0x7f6040198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:58:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f603000d950 tx=0x7f603000dd10 comp rx=0 tx=0).stop 2024-05-26T11:58:57.824 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 shutdown_connections 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f602c0b2900 0x7f602c0b4dc0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6040104040 0x7f604019fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f60401024a0 0x7f6040198560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 --2- 172.21.15.186:0/1394786473 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60401036a0 0x7f6040198aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 >> 172.21.15.186:0/1394786473 conn(0x7f60400fda60 msgr2=0x7f6040106a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 shutdown_connections 2024-05-26T11:58:57.825 INFO:teuthology.orchestra.run.smithi186.stderr:2024-05-26T11:58:57.824+0000 7f6047d9d700 1 -- 172.21.15.186:0/1394786473 wait complete. 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.441 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:58 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:58.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:58 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:58:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-26T11:58:58.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:58:58.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:58 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:58:58.587 DEBUG:teuthology.orchestra.run.smithi186:osd.10> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.10.service 2024-05-26T11:58:58.589 INFO:tasks.cephadm:Deploying osd.11 on smithi192 with /dev/vg_nvme/lv_1... 2024-05-26T11:58:58.590 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-26T11:58:58.835 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:58:59.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:59 smithi156 ceph-mon[32025]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-26T11:58:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:59 smithi156 ceph-mon[32025]: osdmap e63: 11 total, 10 up, 11 in 2024-05-26T11:58:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:59 smithi156 ceph-mon[32025]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:58:59 smithi156 ceph-mon[32025]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:59 smithi192 ceph-mon[36545]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-26T11:58:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:59 smithi192 ceph-mon[36545]: osdmap e63: 11 total, 10 up, 11 in 2024-05-26T11:58:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:59 smithi192 ceph-mon[36545]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:58:59 smithi192 ceph-mon[36545]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.888 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:59 smithi186 ceph-mon[36412]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-26T11:58:59.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:59 smithi186 ceph-mon[36412]: osdmap e63: 11 total, 10 up, 11 in 2024-05-26T11:58:59.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:58:59.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:59 smithi186 ceph-mon[36412]: from='osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:58:59 smithi186 ceph-mon[36412]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]: dispatch 2024-05-26T11:58:59.889 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 11:58:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T11:58:59.537+0000 7f454534f700 -1 osd.10 0 waiting for initial osdmap 2024-05-26T11:58:59.889 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 11:58:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T11:58:59.551+0000 7f4541142700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:59:00.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: osdmap e64: 11 total, 10 up, 11 in 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] boot 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: osdmap e65: 11 total, 11 up, 11 in 2024-05-26T11:59:00.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:00 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: osdmap e64: 11 total, 10 up, 11 in 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] boot 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: osdmap e65: 11 total, 11 up, 11 in 2024-05-26T11:59:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:00 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10+0 records in 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr:10+0 records out 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0110272 s, 951 MB/s 2024-05-26T11:59:00.956 INFO:teuthology.orchestra.run.smithi192.stderr:--> Zapping successful for: 2024-05-26T11:59:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 282 MiB used, 894 GiB / 894 GiB avail 2024-05-26T11:59:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi186", "root=default"]}]': finished 2024-05-26T11:59:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: osdmap e64: 11 total, 10 up, 11 in 2024-05-26T11:59:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:00.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: osd.10 [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] boot 2024-05-26T11:59:00.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: osdmap e65: 11 total, 11 up, 11 in 2024-05-26T11:59:00.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:00 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-26T11:59:01.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:01 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:59:01.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:01 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:59:01.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:01 smithi186 ceph-mon[36412]: osdmap e66: 11 total, 11 up, 11 in 2024-05-26T11:59:01.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:01 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:59:01.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:01 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:59:01.767 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:01 smithi192 ceph-mon[36545]: osdmap e66: 11 total, 11 up, 11 in 2024-05-26T11:59:01.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:01 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:59:01.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:01 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:59:01.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:01 smithi156 ceph-mon[32025]: osdmap e66: 11 total, 11 up, 11 in 2024-05-26T11:59:01.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch daemon add osd smithi192:vg_nvme/lv_1 2024-05-26T11:59:02.043 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:59:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:02.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:02.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:02 smithi192 ceph-mon[36545]: osdmap e67: 11 total, 11 up, 11 in 2024-05-26T11:59:02.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:03.014 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:03.015 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.015 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:02 smithi186 ceph-mon[36412]: osdmap e67: 11 total, 11 up, 11 in 2024-05-26T11:59:03.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:03.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:02 smithi156 ceph-mon[32025]: osdmap e67: 11 total, 11 up, 11 in 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 -- 172.21.15.192:0/982122752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541047a0 msgr2=0x7f3454104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 --2- 172.21.15.192:0/982122752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541047a0 0x7f3454104c00 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3444009990 tx=0x7f3444009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 -- 172.21.15.192:0/982122752 shutdown_connections 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 --2- 172.21.15.192:0/982122752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 0x7f345410b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 --2- 172.21.15.192:0/982122752 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541035a0 0x7f34541039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 --2- 172.21.15.192:0/982122752 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541047a0 0x7f3454104c00 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 -- 172.21.15.192:0/982122752 >> 172.21.15.192:0/982122752 conn(0x7f34540feb40 msgr2=0x7f3454100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 -- 172.21.15.192:0/982122752 shutdown_connections 2024-05-26T11:59:03.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.691+0000 7f34596a5700 1 -- 172.21.15.192:0/982122752 wait complete. 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 Processor -- start 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 -- start start 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541047a0 0x7f3454199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 0x7f34541a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:03.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f345410f090 con 0x7f34541047a0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f345410ef10 con 0x7f3454105140 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34596a5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f345410f210 con 0x7f34541035a0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:36210/0 (socket says 172.21.15.192:36210) 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 -- 172.21.15.192:0/3563931257 learned_addr learned my addr 172.21.15.192:0/3563931257 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34527fc700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541047a0 0x7f3454199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f34537fe700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 0x7f34541a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 -- 172.21.15.192:0/3563931257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 msgr2=0x7f34541a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 0x7f34541a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 -- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541047a0 msgr2=0x7f3454199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.692+0000 7f3452ffd700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541047a0 0x7f3454199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f3452ffd700 1 -- 172.21.15.192:0/3563931257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34541a1390 con 0x7f34541035a0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f3452ffd700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c9a0 tx=0x7f343c00ccb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:03.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c004220 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34541a1680 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f343c007d10 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c003d20 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.693+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34541a1c10 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.694+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f345406a230 con 0x7f34541035a0 2024-05-26T11:59:03.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.695+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f343c007860 con 0x7f34541035a0 2024-05-26T11:59:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.696+0000 7f344bfff700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 0x7f34400b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.696+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7f343c0e1be0 con 0x7f34541035a0 2024-05-26T11:59:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.696+0000 7f34527fc700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 0x7f34400b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:03.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.696+0000 7f34527fc700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 0x7f34400b4d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f344400b5c0 tx=0x7f3444005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:03.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.699+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f343c014030 con 0x7f34541035a0 2024-05-26T11:59:03.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:03 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi186 2024-05-26T11:59:03.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:03 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-26T11:59:03.914 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:03.913+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3454063980 con 0x7f34400b2880 2024-05-26T11:59:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:03 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi186 2024-05-26T11:59:03.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:03 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-26T11:59:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:03 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi186 2024-05-26T11:59:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:03 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi186 to 4272M 2024-05-26T11:59:04.783 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:04 smithi192 ceph-mon[36545]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:04.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:59:04.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:59:04.784 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:04 smithi186 ceph-mon[36412]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:59:04.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:59:04.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:05.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:04 smithi156 ceph-mon[32025]: pgmap v235: 1 pgs: 1 peering; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-26T11:59:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-26T11:59:05.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:05.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:05 smithi192 ceph-mon[36545]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:05.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:05 smithi186 ceph-mon[36412]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:06.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:05 smithi156 ceph-mon[32025]: from='client.24349 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi192:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:06 smithi192 ceph-mon[36545]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:06.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:06 smithi186 ceph-mon[36412]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:07.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:06 smithi156 ceph-mon[32025]: pgmap v236: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/3700158394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]': finished 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: osdmap e68: 12 total, 11 up, 12 in 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:07 smithi192 ceph-mon[36545]: from='client.? 172.21.15.192:0/2061443544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:59:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/3700158394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]': finished 2024-05-26T11:59:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: osdmap e68: 12 total, 11 up, 12 in 2024-05-26T11:59:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:07.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:07 smithi186 ceph-mon[36412]: from='client.? 172.21.15.192:0/2061443544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:59:08.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/3700158394' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:08.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]: dispatch 2024-05-26T11:59:08.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6330d810-016e-4e48-b4ca-4a71ae7317ff"}]': finished 2024-05-26T11:59:08.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: osdmap e68: 12 total, 11 up, 12 in 2024-05-26T11:59:08.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:08.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:07 smithi156 ceph-mon[32025]: from='client.? 172.21.15.192:0/2061443544' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-26T11:59:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:08 smithi192 ceph-mon[36545]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:08.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:08 smithi186 ceph-mon[36412]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:09.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:08 smithi156 ceph-mon[32025]: pgmap v237: 1 pgs: 1 peering; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:10.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:10 smithi192 ceph-mon[36545]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:59:10.840 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:10 smithi156 ceph-mon[32025]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:59:10.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:10 smithi186 ceph-mon[36412]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-05-26T11:59:12.732 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:12 smithi192 ceph-mon[36545]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-26T11:59:12.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:12 smithi186 ceph-mon[36412]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-26T11:59:13.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:12 smithi156 ceph-mon[32025]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-05-26T11:59:14.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:14 smithi186 ceph-mon[36412]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:15.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:14 smithi156 ceph-mon[32025]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:15.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:14 smithi192 ceph-mon[36545]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:15.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-26T11:59:15.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:15 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:16.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-26T11:59:16.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:15 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:16.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-26T11:59:16.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:15 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:16.878 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:16 smithi186 ceph-mon[36412]: Deploying daemon osd.11 on smithi192 2024-05-26T11:59:16.879 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:16 smithi186 ceph-mon[36412]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:16.931 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:16 smithi192 ceph-mon[36545]: Deploying daemon osd.11 on smithi192 2024-05-26T11:59:16.932 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:16 smithi192 ceph-mon[36545]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:17.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:16 smithi156 ceph-mon[32025]: Deploying daemon osd.11 on smithi192 2024-05-26T11:59:17.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:16 smithi156 ceph-mon[32025]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:18 smithi192 ceph-mon[36545]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:18.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:18 smithi186 ceph-mon[36412]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:19.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:18 smithi156 ceph-mon[32025]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-26T11:59:20.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:20 smithi186 ceph-mon[36412]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-26T11:59:20.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:20.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:20.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:20.981 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:20 smithi192 ceph-mon[36545]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-26T11:59:20.981 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:20.981 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:20.982 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:21.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:20 smithi156 ceph-mon[32025]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-05-26T11:59:21.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:21.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:21.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:22 smithi192 ceph-mon[36545]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:22.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:22 smithi186 ceph-mon[36412]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:23.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:22 smithi156 ceph-mon[32025]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:24.131 INFO:teuthology.orchestra.run.smithi192.stdout:Created osd(s) 11 on host 'smithi192' 2024-05-26T11:59:24.131 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.129+0000 7f344bfff700 1 -- 172.21.15.192:0/3563931257 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3454063980 con 0x7f34400b2880 2024-05-26T11:59:24.133 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 msgr2=0x7f34400b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:24.133 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 0x7f34400b4d40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f344400b5c0 tx=0x7f3444005fb0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.133 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 msgr2=0x7f3454199730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f343c00c9a0 tx=0x7f343c00ccb0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 shutdown_connections 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34400b2880 0x7f34400b4d40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3454105140 0x7f34541a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f34541035a0 0x7f3454199730 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 --2- 172.21.15.192:0/3563931257 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f34541047a0 0x7f3454199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 >> 172.21.15.192:0/3563931257 conn(0x7f34540feb40 msgr2=0x7f3454107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 shutdown_connections 2024-05-26T11:59:24.134 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:24.132+0000 7f34596a5700 1 -- 172.21.15.192:0/3563931257 wait complete. 2024-05-26T11:59:24.735 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:24 smithi192 ceph-mon[36545]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:24.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:24.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:24.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:24 smithi192 ceph-mon[36545]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:24.736 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:24 smithi192 ceph-mon[36545]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:24.762 DEBUG:teuthology.orchestra.run.smithi192:osd.11> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.11.service 2024-05-26T11:59:24.765 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-26T11:59:24.765 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-26T11:59:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:24 smithi186 ceph-mon[36412]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:24 smithi186 ceph-mon[36412]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:24.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:24 smithi186 ceph-mon[36412]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:25.021 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:24 smithi156 ceph-mon[32025]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:24 smithi156 ceph-mon[32025]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:25.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:24 smithi156 ceph-mon[32025]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-26T11:59:26.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: osdmap e69: 12 total, 11 up, 12 in 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:25 smithi186 ceph-mon[36412]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.227 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-26T11:59:26.228 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: osdmap e69: 12 total, 11 up, 12 in 2024-05-26T11:59:26.228 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:26.228 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.228 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:25 smithi156 ceph-mon[32025]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: osdmap e69: 12 total, 11 up, 12 in 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:25 smithi192 ceph-mon[36545]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]: dispatch 2024-05-26T11:59:26.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 -- 172.21.15.156:0/3123117896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0070109300 msgr2=0x7f007010b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:26.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 --2- 172.21.15.156:0/3123117896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0070109300 0x7f007010b710 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f006800b0a0 tx=0x7f006800b3b0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 -- 172.21.15.156:0/3123117896 shutdown_connections 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 --2- 172.21.15.156:0/3123117896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f0070109300 0x7f007010b710 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 --2- 172.21.15.156:0/3123117896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 0x7f0070106490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 --2- 172.21.15.156:0/3123117896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00701069d0 0x7f0070108dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 -- 172.21.15.156:0/3123117896 >> 172.21.15.156:0/3123117896 conn(0x7f00700fda70 msgr2=0x7f00700ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 -- 172.21.15.156:0/3123117896 shutdown_connections 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.732+0000 7f0077bab700 1 -- 172.21.15.156:0/3123117896 wait complete. 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 Processor -- start 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 -- start start 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 0x7f0070199680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0070109300 0x7f00701a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f007010f010 con 0x7f00701069d0 2024-05-26T11:59:26.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f007010ee90 con 0x7f0070109300 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0077bab700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f007010f190 con 0x7f00701040a0 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0075146700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0075146700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50082/0 (socket says 172.21.15.156:50082) 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0075146700 1 -- 172.21.15.156:0/3803015353 learned_addr learned my addr 172.21.15.156:0/3803015353 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.733+0000 7f0075947700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 0x7f0070199680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0076148700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0070109300 0x7f00701a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 -- 172.21.15.156:0/3803015353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 msgr2=0x7f0070199680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 0x7f0070199680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.737 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 -- 172.21.15.156:0/3803015353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0070109300 msgr2=0x7f00701a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0070109300 0x7f00701a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 -- 172.21.15.156:0/3803015353 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00701a1290 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0075146700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f006000c950 tx=0x7f006000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060007ac0 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0060004ce0 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0060005580 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00701a1580 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.734+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00701a1b10 con 0x7f00701069d0 2024-05-26T11:59:26.738 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.735+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f007006a230 con 0x7f00701069d0 2024-05-26T11:59:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.737+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f0060007c60 con 0x7f00701069d0 2024-05-26T11:59:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.737+0000 7f0066ffd700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 0x7f005c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.737+0000 7f0075947700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 0x7f005c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.737+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(70..70 src has 1..70) ==== 6847+0+0 (secure 0 0 0) 0x7f00600e1b60 con 0x7f00701069d0 2024-05-26T11:59:26.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.738+0000 7f0075947700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 0x7f005c0b4dc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f006c00ac30 tx=0x7f006c00a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:26.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.741+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f0060014030 con 0x7f00701069d0 2024-05-26T11:59:26.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 11:59:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T11:59:26.543+0000 7fd686efa700 -1 osd.11 0 waiting for initial osdmap 2024-05-26T11:59:26.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 11:59:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T11:59:26.556+0000 7fd67ece5700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-26T11:59:26.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.946+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f007002e590 con 0x7f00701069d0 2024-05-26T11:59:26.947 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.947+0000 7f0066ffd700 1 -- 172.21.15.156:0/3803015353 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v70) ==== 74+0+133 (secure 0 0 0) 0x7f00600a9ec0 con 0x7f00701069d0 2024-05-26T11:59:26.947 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 msgr2=0x7f005c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 0x7f005c0b4dc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f006c00ac30 tx=0x7f006c00a380 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 msgr2=0x7f0070199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f006000c950 tx=0x7f006000cc60 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 shutdown_connections 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f005c0b2900 0x7f005c0b4dc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0070109300 0x7f00701a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f00701040a0 0x7f0070199680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 --2- 172.21.15.156:0/3803015353 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f00701069d0 0x7f0070199bc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:26.950 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 >> 172.21.15.156:0/3803015353 conn(0x7f00700fda70 msgr2=0x7f0070104940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:26.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 shutdown_connections 2024-05-26T11:59:26.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:26.949+0000 7f0077bab700 1 -- 172.21.15.156:0/3803015353 wait complete. 2024-05-26T11:59:27.266 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:59:27.266 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: osdmap e70: 12 total, 11 up, 12 in 2024-05-26T11:59:27.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: from='osd.11 ' entity='osd.11' 2024-05-26T11:59:27.267 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:27 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3803015353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: osdmap e70: 12 total, 11 up, 12 in 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: from='osd.11 ' entity='osd.11' 2024-05-26T11:59:27.413 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:27 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3803015353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:27.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi192", "root=default"]}]': finished 2024-05-26T11:59:27.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: osdmap e70: 12 total, 11 up, 12 in 2024-05-26T11:59:27.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:27.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: from='osd.11 ' entity='osd.11' 2024-05-26T11:59:27.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:27 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3803015353' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:27.655 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":70,"num_osds":12,"num_up_osds":11,"osd_up_since":1716724740,"num_in_osds":12,"osd_in_since":1716724746,"num_remapped_pgs":0} 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: purged_snaps scrub starts 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: purged_snaps scrub ok 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] boot 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: osdmap e71: 12 total, 12 up, 12 in 2024-05-26T11:59:28.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: purged_snaps scrub starts 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: purged_snaps scrub ok 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] boot 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: osdmap e71: 12 total, 12 up, 12 in 2024-05-26T11:59:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: purged_snaps scrub starts 2024-05-26T11:59:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: purged_snaps scrub ok 2024-05-26T11:59:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-26T11:59:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: osd.11 [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] boot 2024-05-26T11:59:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: osdmap e71: 12 total, 12 up, 12 in 2024-05-26T11:59:28.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-26T11:59:28.656 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd stat -f json 2024-05-26T11:59:28.906 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:29.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: Detected new or changed devices on smithi192 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: Adjusting osd_memory_target on smithi192 to 4272M 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:29.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: Detected new or changed devices on smithi192 2024-05-26T11:59:29.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: Adjusting osd_memory_target on smithi192 to 4272M 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: Detected new or changed devices on smithi192 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: Adjusting osd_memory_target on smithi192 to 4272M 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:29.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 -- 172.21.15.156:0/354941701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744104030 msgr2=0x7ff74410a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 --2- 172.21.15.156:0/354941701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744104030 0x7ff74410a8e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff73c00b0d0 tx=0x7ff73c00b3e0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 -- 172.21.15.156:0/354941701 shutdown_connections 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 --2- 172.21.15.156:0/354941701 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744104030 0x7ff74410a8e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 --2- 172.21.15.156:0/354941701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744102490 0x7ff7441028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 --2- 172.21.15.156:0/354941701 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744103690 0x7ff744103af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 -- 172.21.15.156:0/354941701 >> 172.21.15.156:0/354941701 conn(0x7ff7440fda30 msgr2=0x7ff7440ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:30.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 -- 172.21.15.156:0/354941701 shutdown_connections 2024-05-26T11:59:30.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.586+0000 7ff74b562700 1 -- 172.21.15.156:0/354941701 wait complete. 2024-05-26T11:59:30.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 Processor -- start 2024-05-26T11:59:30.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 -- start start 2024-05-26T11:59:30.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 0x7ff744199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:30.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 0x7ff7441a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410df10 con 0x7ff744103690 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410dd90 con 0x7ff744104030 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff74b562700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff74410e090 con 0x7ff744102490 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50112/0 (socket says 172.21.15.156:50112) 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 -- 172.21.15.156:0/1216617223 learned_addr learned my addr 172.21.15.156:0/1216617223 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff7492fe700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 0x7ff744199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff749aff700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 0x7ff7441a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 -- 172.21.15.156:0/1216617223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 msgr2=0x7ff744199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 0x7ff744199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 -- 172.21.15.156:0/1216617223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 msgr2=0x7ff7441a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 0x7ff7441a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 -- 172.21.15.156:0/1216617223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7441a12d0 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff7492fe700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 0x7ff744199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff749aff700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 0x7ff7441a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff748afd700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff74000d9b0 tx=0x7ff74000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.587+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff740016070 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.588+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7441a15c0 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.588+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff7400044d0 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.588+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff740004d30 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.588+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7441a1b50 con 0x7ff744103690 2024-05-26T11:59:30.591 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.588+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff74406a230 con 0x7ff744103690 2024-05-26T11:59:30.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.590+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7ff74000bb60 con 0x7ff744103690 2024-05-26T11:59:30.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.590+0000 7ff7367fc700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 0x7ff7300b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:30.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.591+0000 7ff7492fe700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 0x7ff7300b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:30.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.591+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff7400e2140 con 0x7ff744103690 2024-05-26T11:59:30.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.591+0000 7ff7492fe700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 0x7ff7300b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7441034f0 tx=0x7ff738005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:30.594 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.594+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff74001f020 con 0x7ff744103690 2024-05-26T11:59:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:30 smithi186 ceph-mon[36412]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:30 smithi186 ceph-mon[36412]: osdmap e72: 12 total, 12 up, 12 in 2024-05-26T11:59:30.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:30 smithi156 ceph-mon[32025]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:30.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:30 smithi156 ceph-mon[32025]: osdmap e72: 12 total, 12 up, 12 in 2024-05-26T11:59:30.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.797+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7ff7441a1ee0 con 0x7ff744103690 2024-05-26T11:59:30.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.798+0000 7ff7367fc700 1 -- 172.21.15.156:0/1216617223 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7ff7400aa300 con 0x7ff744103690 2024-05-26T11:59:30.798 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 msgr2=0x7ff7300b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 0x7ff7300b4d40 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff7441034f0 tx=0x7ff738005f90 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 msgr2=0x7ff744199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff74000d9b0 tx=0x7ff74000dd70 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 shutdown_connections 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff7300b2880 0x7ff7300b4d40 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff744104030 0x7ff7441a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff744102490 0x7ff744199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 --2- 172.21.15.156:0/1216617223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff744103690 0x7ff744199bb0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 >> 172.21.15.156:0/1216617223 conn(0x7ff7440fda30 msgr2=0x7ff7440ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:30.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 shutdown_connections 2024-05-26T11:59:30.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:30.800+0000 7ff74b562700 1 -- 172.21.15.156:0/1216617223 wait complete. 2024-05-26T11:59:30.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:30 smithi192 ceph-mon[36545]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 341 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:30.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:30 smithi192 ceph-mon[36545]: osdmap e72: 12 total, 12 up, 12 in 2024-05-26T11:59:31.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:31 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1216617223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:31.670 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1716724767,"num_in_osds":12,"osd_in_since":1716724746,"num_remapped_pgs":0} 2024-05-26T11:59:31.670 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T11:59:31.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:31 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1216617223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:31.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:31 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1216617223' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-26T11:59:31.914 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:32.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:32 smithi186 ceph-mon[36412]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:32.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:32 smithi156 ceph-mon[32025]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:32.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:32 smithi192 ceph-mon[36545]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:33.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 -- 172.21.15.156:0/1710260131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c108920 msgr2=0x7f619c1104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:33.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 --2- 172.21.15.156:0/1710260131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c108920 0x7f619c1104a0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f619400b0d0 tx=0x7f619400b3e0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 -- 172.21.15.156:0/1710260131 shutdown_connections 2024-05-26T11:59:33.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 --2- 172.21.15.156:0/1710260131 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c108920 0x7f619c1104a0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 --2- 172.21.15.156:0/1710260131 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c103850 0x7f619c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 --2- 172.21.15.156:0/1710260131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c104200 0x7f619c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 -- 172.21.15.156:0/1710260131 >> 172.21.15.156:0/1710260131 conn(0x7f619c07a620 msgr2=0x7f619c07aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:33.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 -- 172.21.15.156:0/1710260131 shutdown_connections 2024-05-26T11:59:33.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.445+0000 7f61a121e700 1 -- 172.21.15.156:0/1710260131 wait complete. 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.446+0000 7f61a121e700 1 Processor -- start 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.446+0000 7f61a121e700 1 -- start start 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c104200 0x7f619c19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 0x7f619c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f619c1137c0 con 0x7f619c103850 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f619c113640 con 0x7f619c108920 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f61a121e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f619c113940 con 0x7f619c104200 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:33.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:50130/0 (socket says 172.21.15.156:50130) 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 -- 172.21.15.156:0/3743279241 learned_addr learned my addr 172.21.15.156:0/3743279241 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619b59e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 0x7f619c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619a59c700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c104200 0x7f619c19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 -- 172.21.15.156:0/3743279241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c104200 msgr2=0x7f619c19c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c104200 0x7f619c19c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 -- 172.21.15.156:0/3743279241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 msgr2=0x7f619c1a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 0x7f619c1a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 -- 172.21.15.156:0/3743279241 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f619c1a0de0 con 0x7f619c103850 2024-05-26T11:59:33.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619b59e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 0x7f619c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.447+0000 7f619ad9d700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f618c00b530 tx=0x7f618c00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.448+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c015020 con 0x7f619c103850 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.448+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f618c011ce0 con 0x7f619c103850 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.448+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f619c1a10d0 con 0x7f619c103850 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.448+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f618c010dd0 con 0x7f619c103850 2024-05-26T11:59:33.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.448+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f619c1a8950 con 0x7f619c103850 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.449+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f619c06a230 con 0x7f619c103850 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.450+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f618c018020 con 0x7f619c103850 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.451+0000 7f6183fff700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 0x7f61840b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.451+0000 7f619a59c700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 0x7f61840b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.451+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f618c0e1a30 con 0x7f619c103850 2024-05-26T11:59:33.452 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.451+0000 7f619a59c700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 0x7f61840b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f619c110850 tx=0x7f619000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:33.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.455+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f618c01f020 con 0x7f619c103850 2024-05-26T11:59:33.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.660+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f619c1a8be0 con 0x7f619c103850 2024-05-26T11:59:33.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.661+0000 7f6183fff700 1 -- 172.21.15.156:0/3743279241 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18502 (secure 0 0 0) 0x7f618c0a9bf0 con 0x7f619c103850 2024-05-26T11:59:33.662 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:33.662 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":72,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T11:59:29.421333+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 msgr2=0x7f61840b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 0x7f61840b4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f619c110850 tx=0x7f619000a300 comp rx=0 tx=0).stop 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 msgr2=0x7f619c19bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f618c00b530 tx=0x7f618c00b840 comp rx=0 tx=0).stop 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 shutdown_connections 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61840b2880 0x7f61840b4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f619c108920 0x7f619c1a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f619c103850 0x7f619c19bdd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.664 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 --2- 172.21.15.156:0/3743279241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f619c104200 0x7f619c19c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:33.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 >> 172.21.15.156:0/3743279241 conn(0x7f619c07a620 msgr2=0x7f619c101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:33.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 shutdown_connections 2024-05-26T11:59:33.665 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:33.663+0000 7f61a121e700 1 -- 172.21.15.156:0/3743279241 wait complete. 2024-05-26T11:59:34.506 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-26T11:55:34.798357+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-26T11:59:34.507 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-26T11:59:34.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:34 smithi186 ceph-mon[36412]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:34.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:34 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3743279241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T11:59:34.749 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:34 smithi156 ceph-mon[32025]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:34.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:34 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3743279241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T11:59:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:34 smithi192 ceph-mon[36545]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:34 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3743279241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 -- 172.21.15.156:0/304306968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac108520 msgr2=0x7f26ac108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/304306968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac108520 0x7f26ac108900 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f26a0009990 tx=0x7f26a0009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 -- 172.21.15.156:0/304306968 shutdown_connections 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/304306968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 0x7f26ac10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/304306968 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac108520 0x7f26ac108900 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/304306968 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac102440 0x7f26ac1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.595 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 -- 172.21.15.156:0/304306968 >> 172.21.15.156:0/304306968 conn(0x7f26ac0793f0 msgr2=0x7f26ac079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 -- 172.21.15.156:0/304306968 shutdown_connections 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.592+0000 7f26b3ffc700 1 -- 172.21.15.156:0/304306968 wait complete. 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 Processor -- start 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 -- start start 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 0x7f26ac19afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac108520 0x7f26ac1a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac1125d0 con 0x7f26ac102440 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac112450 con 0x7f26ac102e00 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b3ffc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f26ac112750 con 0x7f26ac108520 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b1d98700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b1d98700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39694/0 (socket says 172.21.15.156:39694) 2024-05-26T11:59:36.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b1d98700 1 -- 172.21.15.156:0/3438333728 learned_addr learned my addr 172.21.15.156:0/3438333728 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.593+0000 7f26b1597700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 0x7f26ac19afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b2599700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac108520 0x7f26ac1a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 -- 172.21.15.156:0/3438333728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac108520 msgr2=0x7f26ac1a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac108520 0x7f26ac1a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 -- 172.21.15.156:0/3438333728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 msgr2=0x7f26ac19afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 0x7f26ac19afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 -- 172.21.15.156:0/3438333728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ac1a6030 con 0x7f26ac102440 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b1d98700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a00047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a0027070 con 0x7f26ac102440 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f26a00041d0 con 0x7f26ac102440 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ac1a6200 con 0x7f26ac102440 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26a000f460 con 0x7f26ac102440 2024-05-26T11:59:36.597 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.594+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ac1a66b0 con 0x7f26ac102440 2024-05-26T11:59:36.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.595+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ac10a9b0 con 0x7f26ac102440 2024-05-26T11:59:36.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.597+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f26a002a070 con 0x7f26ac102440 2024-05-26T11:59:36.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.598+0000 7f269effd700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 0x7f26980b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:36.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.598+0000 7f26b1597700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 0x7f26980b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:36.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.598+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f26a00e30b0 con 0x7f26ac102440 2024-05-26T11:59:36.599 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.598+0000 7f26b1597700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 0x7f26980b4dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005ea0 tx=0x7f26a8005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:36.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.601+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f26a0031070 con 0x7f26ac102440 2024-05-26T11:59:36.602 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:36 smithi156 ceph-mon[32025]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:36.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:36 smithi186 ceph-mon[36412]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:36.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.805+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f26ac06b810 con 0x7f26ac102440 2024-05-26T11:59:36.806 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.805+0000 7f269effd700 1 -- 172.21.15.156:0/3438333728 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f26a0013070 con 0x7f26ac102440 2024-05-26T11:59:36.806 INFO:teuthology.orchestra.run.smithi156.stdout:pg_num: 1 2024-05-26T11:59:36.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.807+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 msgr2=0x7f26980b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:36.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.807+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 0x7f26980b4dc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f26a8005ea0 tx=0x7f26a8005e30 comp rx=0 tx=0).stop 2024-05-26T11:59:36.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.807+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 msgr2=0x7f26ac19aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:36.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.807+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f26a0000c00 tx=0x7f26a00047b0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 shutdown_connections 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f26980b2900 0x7f26980b4dc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f26ac108520 0x7f26ac1a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f26ac102440 0x7f26ac19aa70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 --2- 172.21.15.156:0/3438333728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26ac102e00 0x7f26ac19afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 >> 172.21.15.156:0/3438333728 conn(0x7f26ac0793f0 msgr2=0x7f26ac101240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 shutdown_connections 2024-05-26T11:59:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:36.808+0000 7f26b3ffc700 1 -- 172.21.15.156:0/3438333728 wait complete. 2024-05-26T11:59:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:36 smithi192 ceph-mon[36545]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:37.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:37 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3438333728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-26T11:59:37.620 INFO:tasks.cephadm:Adding mds.a on smithi156 2024-05-26T11:59:37.620 INFO:tasks.cephadm:Adding mds.d on smithi156 2024-05-26T11:59:37.620 INFO:tasks.cephadm:Adding mds.g on smithi156 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.j on smithi156 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.b on smithi186 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.e on smithi186 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.h on smithi186 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.k on smithi186 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.c on smithi192 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.f on smithi192 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.i on smithi192 2024-05-26T11:59:37.621 INFO:tasks.cephadm:Adding mds.l on smithi192 2024-05-26T11:59:37.621 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph orch apply mds all '12;smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l' 2024-05-26T11:59:37.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:37 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3438333728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-26T11:59:37.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:37 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3438333728' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-26T11:59:37.870 INFO:teuthology.orchestra.run.smithi192.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.c/config 2024-05-26T11:59:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:38 smithi186 ceph-mon[36412]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:38.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:38 smithi156 ceph-mon[32025]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:38 smithi192 ceph-mon[36545]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 -- 172.21.15.192:0/2412525264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 msgr2=0x7fee8c103af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 --2- 172.21.15.192:0/2412525264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c103af0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fee80009990 tx=0x7fee80009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 -- 172.21.15.192:0/2412525264 shutdown_connections 2024-05-26T11:59:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 --2- 172.21.15.192:0/2412525264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c104030 0x7fee8c10a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.480 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 --2- 172.21.15.192:0/2412525264 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c102490 0x7fee8c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 --2- 172.21.15.192:0/2412525264 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c103af0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 -- 172.21.15.192:0/2412525264 >> 172.21.15.192:0/2412525264 conn(0x7fee8c0fda30 msgr2=0x7fee8c0ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 -- 172.21.15.192:0/2412525264 shutdown_connections 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.479+0000 7fee929c6700 1 -- 172.21.15.192:0/2412525264 wait complete. 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 Processor -- start 2024-05-26T11:59:39.481 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 -- start start 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c102490 0x7fee8c199730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c104030 0x7fee8c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c10df80 con 0x7fee8c104030 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c10de00 con 0x7fee8c102490 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee929c6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fee8c10e100 con 0x7fee8c103690 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.192:54588/0 (socket says 172.21.15.192:54588) 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 -- 172.21.15.192:0/4074135011 learned_addr learned my addr 172.21.15.192:0/4074135011 (peer_addr_for_me v2:172.21.15.192:0/0) 2024-05-26T11:59:39.483 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8bfff700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c102490 0x7fee8c199730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee90f63700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c104030 0x7fee8c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 -- 172.21.15.192:0/4074135011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c102490 msgr2=0x7fee8c199730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c102490 0x7fee8c199730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 -- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c104030 msgr2=0x7fee8c1a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c104030 0x7fee8c1a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.480+0000 7fee8b7fe700 1 -- 172.21.15.192:0/4074135011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee8c1a1390 con 0x7fee8c103690 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee8b7fe700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee800036f0 tx=0x7fee80009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee80027070 con 0x7fee8c103690 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee8c1a1620 con 0x7fee8c103690 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fee80022470 con 0x7fee8c103690 2024-05-26T11:59:39.484 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee8000f6c0 con 0x7fee8c103690 2024-05-26T11:59:39.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.481+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee8c1a1a80 con 0x7fee8c103690 2024-05-26T11:59:39.485 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.482+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee8c06a230 con 0x7fee8c103690 2024-05-26T11:59:39.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.483+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fee8002a070 con 0x7fee8c103690 2024-05-26T11:59:39.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.484+0000 7fee897fa700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 0x7fee740b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:39.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.484+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fee800e2fe0 con 0x7fee8c103690 2024-05-26T11:59:39.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.484+0000 7fee8bfff700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 0x7fee740b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:39.486 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.484+0000 7fee8bfff700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 0x7fee740b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee8c1034f0 tx=0x7fee7c00b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:39.488 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.487+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fee80013070 con 0x7fee8c103690 2024-05-26T11:59:39.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.700+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}) -- 0x7fee8c06c1a0 con 0x7fee740b2880 2024-05-26T11:59:39.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.708+0000 7fee897fa700 1 -- 172.21.15.192:0/4074135011 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fee8c06c1a0 con 0x7fee740b2880 2024-05-26T11:59:39.709 INFO:teuthology.orchestra.run.smithi192.stdout:Scheduled mds.all update... 2024-05-26T11:59:39.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.710+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 msgr2=0x7fee740b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:39.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.710+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 0x7fee740b4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fee8c1034f0 tx=0x7fee7c00b5b0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.710+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 msgr2=0x7fee8c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:39.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.710+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fee800036f0 tx=0x7fee80009f90 comp rx=0 tx=0).stop 2024-05-26T11:59:39.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.710+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 shutdown_connections 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fee740b2880 0x7fee740b4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fee8c104030 0x7fee8c1a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee8c102490 0x7fee8c199730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 --2- 172.21.15.192:0/4074135011 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fee8c103690 0x7fee8c199c70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 >> 172.21.15.192:0/4074135011 conn(0x7fee8c0fda30 msgr2=0x7fee8c06ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 shutdown_connections 2024-05-26T11:59:39.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-05-26T11:59:39.711+0000 7fee929c6700 1 -- 172.21.15.192:0/4074135011 wait complete. 2024-05-26T11:59:40.342 DEBUG:teuthology.orchestra.run.smithi156:mds.a> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.a.service 2024-05-26T11:59:40.345 DEBUG:teuthology.orchestra.run.smithi156:mds.d> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.d.service 2024-05-26T11:59:40.389 DEBUG:teuthology.orchestra.run.smithi156:mds.g> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.g.service 2024-05-26T11:59:40.432 DEBUG:teuthology.orchestra.run.smithi156:mds.j> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.j.service 2024-05-26T11:59:40.475 DEBUG:teuthology.orchestra.run.smithi186:mds.b> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.b.service 2024-05-26T11:59:40.478 DEBUG:teuthology.orchestra.run.smithi186:mds.e> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.e.service 2024-05-26T11:59:40.521 DEBUG:teuthology.orchestra.run.smithi186:mds.h> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.h.service 2024-05-26T11:59:40.564 DEBUG:teuthology.orchestra.run.smithi186:mds.k> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.k.service 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:40.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.608 DEBUG:teuthology.orchestra.run.smithi192:mds.c> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.c.service 2024-05-26T11:59:40.610 DEBUG:teuthology.orchestra.run.smithi192:mds.f> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.f.service 2024-05-26T11:59:40.653 DEBUG:teuthology.orchestra.run.smithi192:mds.i> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.i.service 2024-05-26T11:59:40.696 DEBUG:teuthology.orchestra.run.smithi192:mds.l> sudo journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.l.service 2024-05-26T11:59:40.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:40.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.739 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-26T11:59:40.739 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T11:59:40.763 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:40.763 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:40.764 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:40.980 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:41.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:41 smithi192 ceph-mon[36545]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:41.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:41 smithi192 ceph-mon[36545]: Saving service mds.all spec with placement smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-05-26T11:59:41.567 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:41 smithi192 ceph-mon[36545]: Deploying daemon mds.l on smithi192 2024-05-26T11:59:41.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:41 smithi186 ceph-mon[36412]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:41.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:41 smithi186 ceph-mon[36412]: Saving service mds.all spec with placement smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-05-26T11:59:41.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:41 smithi186 ceph-mon[36412]: Deploying daemon mds.l on smithi192 2024-05-26T11:59:41.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:41 smithi156 ceph-mon[32025]: from='client.24391 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T11:59:41.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:41 smithi156 ceph-mon[32025]: Saving service mds.all spec with placement smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-05-26T11:59:41.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:41 smithi156 ceph-mon[32025]: Deploying daemon mds.l on smithi192 2024-05-26T11:59:42.240 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 systemd[1]: Starting Ceph mds.l for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:59:42.491 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:42 smithi192 ceph-mon[36545]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:42.491 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 podman[66185]: 2024-05-26 11:59:42.142544016 +0000 UTC m=+0.016836075 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T11:59:42.491 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 podman[66185]: 2024-05-26 11:59:42.254773147 +0000 UTC m=+0.129065234 container create ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-26T11:59:42.491 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 podman[66185]: 2024-05-26 11:59:42.491325233 +0000 UTC m=+0.365617277 container init ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-26T11:59:42.665 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:42 smithi156 ceph-mon[32025]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:42.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:42 smithi186 ceph-mon[36412]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:42.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 -- 172.21.15.156:0/1677254151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773806bcb0 msgr2=0x7f773806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:42.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 --2- 172.21.15.156:0/1677254151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773806bcb0 0x7f773806c130 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f772c009990 tx=0x7f772c009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.773 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 -- 172.21.15.156:0/1677254151 shutdown_connections 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 --2- 172.21.15.156:0/1677254151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f7738110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 --2- 172.21.15.156:0/1677254151 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f77381095e0 0x7f77381099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 --2- 172.21.15.156:0/1677254151 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f773806bcb0 0x7f773806c130 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 -- 172.21.15.156:0/1677254151 >> 172.21.15.156:0/1677254151 conn(0x7f77380784b0 msgr2=0x7f77380788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 -- 172.21.15.156:0/1677254151 shutdown_connections 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.769+0000 7f773dccd700 1 -- 172.21.15.156:0/1677254151 wait complete. 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 Processor -- start 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 -- start start 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 0x7f773819f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f773819fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:42.774 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7738113700 con 0x7f77381095e0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7738113580 con 0x7f773806c670 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.770+0000 7f773dccd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7738113880 con 0x7f773806bcb0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:39714/0 (socket says 172.21.15.156:39714) 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7736ffd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f773819fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 -- 172.21.15.156:0/1864148116 learned_addr learned my addr 172.21.15.156:0/1864148116 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f77377fe700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 0x7f773819f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 -- 172.21.15.156:0/1864148116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 msgr2=0x7f773819f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 0x7f773819f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 -- 172.21.15.156:0/1864148116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 msgr2=0x7f773819fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f773819fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 -- 172.21.15.156:0/1864148116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773819a120 con 0x7f77381095e0 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7736ffd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f773819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T11:59:42.775 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f77377fe700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 0x7f773819f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7737fff700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7730007ee0 tx=0x7f773000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f773000dbd0 con 0x7f77381095e0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773819a410 con 0x7f77381095e0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f773000dd70 con 0x7f77381095e0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7730005990 con 0x7f77381095e0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.771+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77381a66d0 con 0x7f77381095e0 2024-05-26T11:59:42.776 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.772+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773806a230 con 0x7f77381095e0 2024-05-26T11:59:42.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.774+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f773001a020 con 0x7f77381095e0 2024-05-26T11:59:42.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.775+0000 7f7734ff9700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 0x7f77200b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:42.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.775+0000 7f77377fe700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 0x7f77200b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:42.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.775+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f77300e1ca0 con 0x7f77381095e0 2024-05-26T11:59:42.777 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.775+0000 7f77377fe700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 0x7f77200b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7728005ea0 tx=0x7f7728005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:42.779 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:42.778+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7730010070 con 0x7f77381095e0 2024-05-26T11:59:42.822 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 podman[66185]: 2024-05-26 11:59:42.494128802 +0000 UTC m=+0.368420866 container start ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521) 2024-05-26T11:59:42.822 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 ceph-mds[66204]: -- 172.21.15.192:0/2539660713 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55dd76229a40 con 0x55dd761fcc00 2024-05-26T11:59:42.822 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: starting mds.l at 2024-05-26T11:59:42.822 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 bash[66185]: ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 2024-05-26T11:59:42.822 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 11:59:42 smithi192 systemd[1]: Started Ceph mds.l for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:59:43.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.025+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f773802e560 con 0x7f77381095e0 2024-05-26T11:59:43.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.026+0000 7f7734ff9700 1 -- 172.21.15.156:0/1864148116 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7f77300a9dc0 con 0x7f77381095e0 2024-05-26T11:59:43.027 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:43.027 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.028+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 msgr2=0x7f77200b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.028+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 0x7f77200b4d40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7728005ea0 tx=0x7f7728005e30 comp rx=0 tx=0).stop 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.028+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 msgr2=0x7f77381999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.028+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7730007ee0 tx=0x7f773000d3b0 comp rx=0 tx=0).stop 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 shutdown_connections 2024-05-26T11:59:43.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f77200b2880 0x7f77200b4d40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f77381095e0 0x7f77381999c0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f773806bcb0 0x7f773819f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 --2- 172.21.15.156:0/1864148116 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f773806c670 0x7f773819fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 >> 172.21.15.156:0/1864148116 conn(0x7f77380784b0 msgr2=0x7f7738101100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 shutdown_connections 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:43.029+0000 7f773dccd700 1 -- 172.21.15.156:0/1864148116 wait complete. 2024-05-26T11:59:43.030 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 2 2024-05-26T11:59:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6832/4217003952,v1:172.21.15.192:6833/4217003952] up:boot 2024-05-26T11:59:43.544 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: fsmap 1 up:standby 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:43.545 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:43 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1864148116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:43.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6832/4217003952,v1:172.21.15.192:6833/4217003952] up:boot 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: fsmap 1 up:standby 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:43.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:43 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1864148116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6832/4217003952,v1:172.21.15.192:6833/4217003952] up:boot 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: fsmap 1 up:standby 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:43.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:43 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1864148116' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:44.662 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:44 smithi186 ceph-mon[36412]: Deploying daemon mds.k on smithi186 2024-05-26T11:59:44.663 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:44 smithi186 ceph-mon[36412]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:44.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:44 smithi156 ceph-mon[32025]: Deploying daemon mds.k on smithi186 2024-05-26T11:59:44.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:44 smithi156 ceph-mon[32025]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:44.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:44 smithi192 ceph-mon[36545]: Deploying daemon mds.k on smithi186 2024-05-26T11:59:44.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:44 smithi192 ceph-mon[36545]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:44.976 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:44 smithi186 systemd[1]: Starting Ceph mds.k for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:59:45.330 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 podman[66580]: 2024-05-26 11:59:44.959171158 +0000 UTC m=+0.017492511 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T11:59:45.330 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 podman[66580]: 2024-05-26 11:59:45.077617035 +0000 UTC m=+0.135938416 container create 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T11:59:45.726 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 podman[66580]: 2024-05-26 11:59:45.330314013 +0000 UTC m=+0.388635348 container init 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 podman[66580]: 2024-05-26 11:59:45.332976333 +0000 UTC m=+0.391297671 container start 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 ceph-mds[66600]: -- 172.21.15.186:0/2077948586 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x5565f0a164e0 con 0x5565f09ed000 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 ceph-mds[66600]: -- 172.21.15.186:0/2077948586 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5565f0a19a40 con 0x5565f09ed000 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: starting mds.k at 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 bash[66580]: 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea 2024-05-26T11:59:45.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 11:59:45 smithi186 systemd[1]: Started Ceph mds.k for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:59:45.859 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T11:59:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: Deploying daemon mds.j on smithi156 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:boot 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: fsmap 2 up:standby 2024-05-26T11:59:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:46 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: Deploying daemon mds.j on smithi156 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:boot 2024-05-26T11:59:46.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: fsmap 2 up:standby 2024-05-26T11:59:46.796 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:46 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:46.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: Deploying daemon mds.j on smithi156 2024-05-26T11:59:46.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:boot 2024-05-26T11:59:46.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: fsmap 2 up:standby 2024-05-26T11:59:46.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:46 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T11:59:47.305 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:48.726 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:48 smithi156 ceph-mon[32025]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:48 smithi186 ceph-mon[36412]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:48 smithi192 ceph-mon[36545]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 -- 172.21.15.156:0/2506250912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4102c40 msgr2=0x7f64b41030c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 --2- 172.21.15.156:0/2506250912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4102c40 0x7f64b41030c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f64a0009990 tx=0x7f64a0009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 -- 172.21.15.156:0/2506250912 shutdown_connections 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 --2- 172.21.15.156:0/2506250912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4103600 0x7f64b410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 --2- 172.21.15.156:0/2506250912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.297 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 --2- 172.21.15.156:0/2506250912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4102c40 0x7f64b41030c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 -- 172.21.15.156:0/2506250912 >> 172.21.15.156:0/2506250912 conn(0x7f64b4078ab0 msgr2=0x7f64b40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 -- 172.21.15.156:0/2506250912 shutdown_connections 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.283+0000 7f64baf15700 1 -- 172.21.15.156:0/2506250912 wait complete. 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 Processor -- start 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 -- start start 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 0x7f64b419f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f64b41125d0 con 0x7f64b4103600 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4112450 con 0x7f64b4102c40 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64baf15700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f64b4112750 con 0x7f64b41084c0 2024-05-26T11:59:49.298 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64b3fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37542/0 (socket says 172.21.15.156:37542) 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64b3fff700 1 -- 172.21.15.156:0/3838691492 learned_addr learned my addr 172.21.15.156:0/3838691492 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.284+0000 7f64b8cb1700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 0x7f64b419f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b94b2700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 -- 172.21.15.156:0/3838691492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 msgr2=0x7f64b41999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 -- 172.21.15.156:0/3838691492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 msgr2=0x7f64b419f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 0x7f64b419f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 -- 172.21.15.156:0/3838691492 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b419a140 con 0x7f64b4103600 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b94b2700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b8cb1700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 0x7f64b419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b3fff700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f64a00057c0 tx=0x7f64a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:49.299 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0027070 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f64a0004020 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b419a3d0 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a0022bf0 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.285+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b41a66f0 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.286+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b406a230 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.287+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f64a002a070 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.288+0000 7f64b1ffb700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 0x7f64a40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.288+0000 7f64b8cb1700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 0x7f64a40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.288+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f64a00e3400 con 0x7f64b4103600 2024-05-26T11:59:49.300 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.288+0000 7f64b8cb1700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 0x7f64a40b4dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64a80059d0 tx=0x7f64a8005920 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:49.301 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.291+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f64a0031020 con 0x7f64b4103600 2024-05-26T11:59:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.563+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f64b419af10 con 0x7f64b4103600 2024-05-26T11:59:49.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.563+0000 7f64b1ffb700 1 -- 172.21.15.156:0/3838691492 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f64b419af10 con 0x7f64b4103600 2024-05-26T11:59:49.564 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:49.564 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 msgr2=0x7f64a40b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 0x7f64a40b4dc0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f64a80059d0 tx=0x7f64a8005920 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 msgr2=0x7f64b419fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f64a00057c0 tx=0x7f64a00057f0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 shutdown_connections 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f64a40b2900 0x7f64a40b4dc0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f64b41084c0 0x7f64b41999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f64b4102c40 0x7f64b419f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 --2- 172.21.15.156:0/3838691492 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f64b4103600 0x7f64b419fd60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 >> 172.21.15.156:0/3838691492 conn(0x7f64b4078ab0 msgr2=0x7f64b41007b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 shutdown_connections 2024-05-26T11:59:49.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:49.566+0000 7f64baf15700 1 -- 172.21.15.156:0/3838691492 wait complete. 2024-05-26T11:59:49.568 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 3 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3838691492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:50.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:50 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3838691492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:50.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:50 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3838691492' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:50.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:50.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:50 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:51.736 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:51 smithi156 ceph-mon[32025]: Deploying daemon mds.d on smithi156 2024-05-26T11:59:51.736 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:51 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6834/1321300121,v1:172.21.15.156:6835/1321300121] up:boot 2024-05-26T11:59:51.736 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:51 smithi156 ceph-mon[32025]: fsmap 3 up:standby 2024-05-26T11:59:51.736 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:51 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-26T11:59:51.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:51 smithi192 ceph-mon[36545]: Deploying daemon mds.d on smithi156 2024-05-26T11:59:51.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:51 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6834/1321300121,v1:172.21.15.156:6835/1321300121] up:boot 2024-05-26T11:59:51.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:51 smithi192 ceph-mon[36545]: fsmap 3 up:standby 2024-05-26T11:59:51.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:51 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-26T11:59:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:51 smithi186 ceph-mon[36412]: Deploying daemon mds.d on smithi156 2024-05-26T11:59:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:51 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6834/1321300121,v1:172.21.15.156:6835/1321300121] up:boot 2024-05-26T11:59:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:51 smithi186 ceph-mon[36412]: fsmap 3 up:standby 2024-05-26T11:59:51.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:51 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-26T11:59:52.294 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 systemd[1]: Starting Ceph mds.d for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:59:52.487 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T11:59:52.691 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:52 smithi156 ceph-mon[32025]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:52.691 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 podman[69620]: 2024-05-26 11:59:52.293868143 +0000 UTC m=+0.017567225 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T11:59:52.691 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 podman[69620]: 2024-05-26 11:59:52.42109717 +0000 UTC m=+0.144796234 container create 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-26T11:59:52.691 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 podman[69620]: 2024-05-26 11:59:52.690975412 +0000 UTC m=+0.414674489 container init 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True) 2024-05-26T11:59:52.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:52 smithi192 ceph-mon[36545]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:52.846 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:52.970 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 podman[69620]: 2024-05-26 11:59:52.694123062 +0000 UTC m=+0.417822126 container start 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.schema-version=1.0) 2024-05-26T11:59:52.970 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 ceph-mds[69669]: -- 172.21.15.156:0/3910547957 <== mon.0 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55eca5443a40 con 0x55eca5416800 2024-05-26T11:59:52.970 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d[69653]: starting mds.d at 2024-05-26T11:59:52.970 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 bash[69620]: 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 2024-05-26T11:59:52.970 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 11:59:52 smithi156 systemd[1]: Started Ceph mds.d for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:59:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:52 smithi186 ceph-mon[36412]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:54.016 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:54.016 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.016 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: Deploying daemon mds.e on smithi186 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6836/2020460154,v1:172.21.15.156:6837/2020460154] up:boot 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: fsmap 4 up:standby 2024-05-26T11:59:54.017 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:53 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-26T11:59:54.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:54.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: Deploying daemon mds.e on smithi186 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6836/2020460154,v1:172.21.15.156:6837/2020460154] up:boot 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: fsmap 4 up:standby 2024-05-26T11:59:54.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:53 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-26T11:59:54.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:54.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: Deploying daemon mds.e on smithi186 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6836/2020460154,v1:172.21.15.156:6837/2020460154] up:boot 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: fsmap 4 up:standby 2024-05-26T11:59:54.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:53 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-26T11:59:54.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 -- 172.21.15.156:0/1950469927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806bcb0 msgr2=0x7efee806c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:54.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 --2- 172.21.15.156:0/1950469927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806bcb0 0x7efee806c130 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efedc009990 tx=0x7efedc009ca0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.533 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 -- 172.21.15.156:0/1950469927 shutdown_connections 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 --2- 172.21.15.156:0/1950469927 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee806c670 0x7efee8110420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 --2- 172.21.15.156:0/1950469927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee81095e0 0x7efee81099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 --2- 172.21.15.156:0/1950469927 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806bcb0 0x7efee806c130 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 -- 172.21.15.156:0/1950469927 >> 172.21.15.156:0/1950469927 conn(0x7efee80784b0 msgr2=0x7efee80788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 -- 172.21.15.156:0/1950469927 shutdown_connections 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.531+0000 7efeecf3a700 1 -- 172.21.15.156:0/1950469927 wait complete. 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 Processor -- start 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 -- start start 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee806bcb0 0x7efee8102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee81095e0 0x7efee8102fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:54.534 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7efee8113760 con 0x7efee806c670 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7efee81135e0 con 0x7efee806bcb0 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efeecf3a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7efee81138e0 con 0x7efee81095e0 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:37610/0 (socket says 172.21.15.156:37610) 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 -- 172.21.15.156:0/2338450804 learned_addr learned my addr 172.21.15.156:0/2338450804 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee659c700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee806bcb0 0x7efee8102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 -- 172.21.15.156:0/2338450804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee81095e0 msgr2=0x7efee8102fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee81095e0 0x7efee8102fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 -- 172.21.15.156:0/2338450804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee806bcb0 msgr2=0x7efee8102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee806bcb0 0x7efee8102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 -- 172.21.15.156:0/2338450804 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee81a8200 con 0x7efee806c670 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.532+0000 7efee5d9b700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efedc0057c0 tx=0x7efedc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.533+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc027070 con 0x7efee806c670 2024-05-26T11:59:54.535 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.533+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efedc004020 con 0x7efee806c670 2024-05-26T11:59:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.533+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efee81a8490 con 0x7efee806c670 2024-05-26T11:59:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.533+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efedc022bf0 con 0x7efee806c670 2024-05-26T11:59:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.533+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efee81a89c0 con 0x7efee806c670 2024-05-26T11:59:54.536 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.534+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efee8104ca0 con 0x7efee806c670 2024-05-26T11:59:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.535+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7efedc02a070 con 0x7efee806c670 2024-05-26T11:59:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.536+0000 7efed77fe700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 0x7efed00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.536+0000 7efee659c700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 0x7efed00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.536+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7efedc0e3410 con 0x7efee806c670 2024-05-26T11:59:54.537 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.536+0000 7efee659c700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 0x7efed00b4dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efed8005950 tx=0x7efed80058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:54.539 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.539+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7efedc031020 con 0x7efee806c670 2024-05-26T11:59:54.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.791+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7efee802e720 con 0x7efee806c670 2024-05-26T11:59:54.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.792+0000 7efed77fe700 1 -- 172.21.15.156:0/2338450804 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3877 (secure 0 0 0) 0x7efedc013070 con 0x7efee806c670 2024-05-26T11:59:54.792 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:54.793 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 msgr2=0x7efed00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 0x7efed00b4dc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7efed8005950 tx=0x7efed80058e0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 msgr2=0x7efee8102a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7efedc0057c0 tx=0x7efedc0057f0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 shutdown_connections 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7efed00b2900 0x7efed00b4dc0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7efee81095e0 0x7efee8102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efee806bcb0 0x7efee8102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 --2- 172.21.15.156:0/2338450804 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7efee806c670 0x7efee8102a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:54.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 >> 172.21.15.156:0/2338450804 conn(0x7efee80784b0 msgr2=0x7efee8101350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:54.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 shutdown_connections 2024-05-26T11:59:54.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:54.794+0000 7efeecf3a700 1 -- 172.21.15.156:0/2338450804 wait complete. 2024-05-26T11:59:54.796 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 5 2024-05-26T11:59:55.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:54 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2338450804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:55.116 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:54 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2338450804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:55.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:54 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2338450804' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T11:59:55.720 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 11:59:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: starting mds.e at 2024-05-26T11:59:55.720 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 11:59:55 smithi186 bash[66895]: 71176291a1a048a2d162141caa0f2675cccb87b3d4ce4ffac351888cb940a9d4 2024-05-26T11:59:55.976 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 11:59:55 smithi186 systemd[1]: Started Ceph mds.e for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:59:55.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:55.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:55 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:56.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:55 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:55 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:57.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:56 smithi192 ceph-mon[36545]: Deploying daemon mds.f on smithi192 2024-05-26T11:59:57.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:56 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:boot 2024-05-26T11:59:57.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:56 smithi192 ceph-mon[36545]: fsmap 5 up:standby 2024-05-26T11:59:57.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:56 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T11:59:57.152 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:56 smithi186 ceph-mon[36412]: Deploying daemon mds.f on smithi192 2024-05-26T11:59:57.153 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:56 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:boot 2024-05-26T11:59:57.153 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:56 smithi186 ceph-mon[36412]: fsmap 5 up:standby 2024-05-26T11:59:57.153 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:56 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T11:59:57.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:56 smithi156 ceph-mon[32025]: Deploying daemon mds.f on smithi192 2024-05-26T11:59:57.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:56 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:boot 2024-05-26T11:59:57.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:56 smithi156 ceph-mon[32025]: fsmap 5 up:standby 2024-05-26T11:59:57.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:56 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T11:59:57.420 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T11:59:57.675 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T11:59:57.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:57 smithi186 ceph-mon[36412]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:58.071 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:57 smithi156 ceph-mon[32025]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:58.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:57 smithi192 ceph-mon[36545]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T11:59:58.072 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:57 smithi192 systemd[1]: Starting Ceph mds.f for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T11:59:58.432 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 podman[66487]: 2024-05-26 11:59:58.037726325 +0000 UTC m=+0.017420494 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T11:59:58.432 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 podman[66487]: 2024-05-26 11:59:58.163312615 +0000 UTC m=+0.143006770 container create 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 podman[66487]: 2024-05-26 11:59:58.4329623 +0000 UTC m=+0.412656456 container init 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521) 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 podman[66487]: 2024-05-26 11:59:58.435716346 +0000 UTC m=+0.415410511 container start 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2) 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 ceph-mds[66507]: -- 172.21.15.192:0/3975043786 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d34a5dfa40 con 0x55d34a5b2c00 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: starting mds.f at 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 bash[66487]: 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 2024-05-26T11:59:58.822 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 11:59:58 smithi192 systemd[1]: Started Ceph mds.f for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T11:59:59.371 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.368+0000 7f810b42f700 1 -- 172.21.15.156:0/3094116349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104103600 msgr2=0x7f810410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.368+0000 7f810b42f700 1 --2- 172.21.15.156:0/3094116349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104103600 0x7f810410f2e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f80fc00b0a0 tx=0x7f80fc00b3b0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 -- 172.21.15.156:0/3094116349 shutdown_connections 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 --2- 172.21.15.156:0/3094116349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8104103600 0x7f810410f2e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 --2- 172.21.15.156:0/3094116349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f81041084c0 0x7f81041088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 --2- 172.21.15.156:0/3094116349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f81041030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 -- 172.21.15.156:0/3094116349 >> 172.21.15.156:0/3094116349 conn(0x7f8104078ab0 msgr2=0x7f81040ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 -- 172.21.15.156:0/3094116349 shutdown_connections 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 -- 172.21.15.156:0/3094116349 wait complete. 2024-05-26T11:59:59.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 Processor -- start 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.369+0000 7f810b42f700 1 -- start start 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f810419e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8104103600 0x7f810419ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8104112650 con 0x7f81041084c0 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81041124d0 con 0x7f8104102c40 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f81041127d0 con 0x7f8104103600 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56536/0 (socket says 172.21.15.156:56536) 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 -- 172.21.15.156:0/1038107200 learned_addr learned my addr 172.21.15.156:0/1038107200 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81089ca700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8104103600 0x7f810419ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:59.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81091cb700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f810419e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 -- 172.21.15.156:0/1038107200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8104103600 msgr2=0x7f810419ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8104103600 0x7f810419ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 -- 172.21.15.156:0/1038107200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 msgr2=0x7f810419e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f810419e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 -- 172.21.15.156:0/1038107200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8104199070 con 0x7f81041084c0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81091cb700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f810419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f81099cc700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0062a0 tx=0x7f80fc004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc021070 con 0x7f81041084c0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80fc007d00 con 0x7f81041084c0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80fc01d450 con 0x7f81041084c0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8104199300 con 0x7f81041084c0 2024-05-26T11:59:59.374 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.370+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81041a55d0 con 0x7f81041084c0 2024-05-26T11:59:59.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.371+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f810406a230 con 0x7f81041084c0 2024-05-26T11:59:59.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.373+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f80fc004000 con 0x7f81041084c0 2024-05-26T11:59:59.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.374+0000 7f80fa7fc700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 0x7f80f00b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T11:59:59.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.374+0000 7f81091cb700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 0x7f80f00b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T11:59:59.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.374+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f80fc00f020 con 0x7f81041084c0 2024-05-26T11:59:59.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.374+0000 7f81091cb700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 0x7f80f00b4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8100005fd0 tx=0x7f8100005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T11:59:59.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.377+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f80fc02a020 con 0x7f81041084c0 2024-05-26T11:59:59.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.636+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f810402e560 con 0x7f81041084c0 2024-05-26T11:59:59.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.636+0000 7f80fa7fc700 1 -- 172.21.15.156:0/1038107200 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5530 (secure 0 0 0) 0x7f80fc0abad0 con 0x7f81041084c0 2024-05-26T11:59:59.637 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T11:59:59.637 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 msgr2=0x7f80f00b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 0x7f80f00b4d40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8100005fd0 tx=0x7f8100005de0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 msgr2=0x7f8104198910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0062a0 tx=0x7f80fc004ed0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 shutdown_connections 2024-05-26T11:59:59.639 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f80f00b2880 0x7f80f00b4d40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f81041084c0 0x7f8104198910 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8104102c40 0x7f810419e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 --2- 172.21.15.156:0/1038107200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8104103600 0x7f810419ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 >> 172.21.15.156:0/1038107200 conn(0x7f8104078ab0 msgr2=0x7f8104100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 shutdown_connections 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T11:59:59.638+0000 7f810b42f700 1 -- 172.21.15.156:0/1038107200 wait complete. 2024-05-26T11:59:59.640 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 7 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: Deploying daemon mds.c on smithi192 2024-05-26T11:59:59.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6834/699682135,v1:172.21.15.192:6835/699682135] up:boot 2024-05-26T11:59:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: fsmap 6 up:standby 2024-05-26T11:59:59.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 11:59:59 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T11:59:59.960 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.960 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.960 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.960 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:59.960 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:59.961 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:59.961 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: Deploying daemon mds.c on smithi192 2024-05-26T11:59:59.961 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6834/699682135,v1:172.21.15.192:6835/699682135] up:boot 2024-05-26T11:59:59.961 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: fsmap 6 up:standby 2024-05-26T11:59:59.961 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 11:59:59 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T11:59:59.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: Deploying daemon mds.c on smithi192 2024-05-26T11:59:59.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6834/699682135,v1:172.21.15.192:6835/699682135] up:boot 2024-05-26T11:59:59.978 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: fsmap 6 up:standby 2024-05-26T11:59:59.978 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 11:59:59 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T12:00:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:00 smithi192 ceph-mon[36545]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:00 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1038107200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:00.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:00 smithi192 ceph-mon[36545]: overall HEALTH_OK 2024-05-26T12:00:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:00 smithi186 ceph-mon[36412]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:00 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1038107200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:00.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:00 smithi186 ceph-mon[36412]: overall HEALTH_OK 2024-05-26T12:00:01.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:00 smithi156 ceph-mon[32025]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:01.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:00 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1038107200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:01.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:00 smithi156 ceph-mon[32025]: overall HEALTH_OK 2024-05-26T12:00:01.572 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:00:01 smithi192 systemd[1]: Started Ceph mds.c for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:02.291 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:00:02.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:02.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: Deploying daemon mds.h on smithi186 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6836/977346748,v1:172.21.15.192:6837/977346748] up:boot 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: fsmap 7 up:standby 2024-05-26T12:00:02.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:02 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:00:02.547 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: Deploying daemon mds.h on smithi186 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6836/977346748,v1:172.21.15.192:6837/977346748] up:boot 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: fsmap 7 up:standby 2024-05-26T12:00:02.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:02 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:00:02.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:02.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.669 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: Deploying daemon mds.h on smithi186 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6836/977346748,v1:172.21.15.192:6837/977346748] up:boot 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: fsmap 7 up:standby 2024-05-26T12:00:02.670 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:02 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:00:03.726 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:03 smithi186 systemd[1]: Starting Ceph mds.h for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:00:04.050 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:03 smithi186 podman[67204]: 2024-05-26 12:00:03.677417433 +0000 UTC m=+0.017777229 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:00:04.051 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:03 smithi186 podman[67204]: 2024-05-26 12:00:03.797688259 +0000 UTC m=+0.138048033 container create b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, ceph=True, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:00:04.051 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 podman[67204]: 2024-05-26 12:00:04.050727247 +0000 UTC m=+0.391087028 container init b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 -- 172.21.15.156:0/165775537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806bc90 msgr2=0x7f4e1806c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 --2- 172.21.15.156:0/165775537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806bc90 0x7f4e1806c110 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c0099c0 tx=0x7f4e0c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 -- 172.21.15.156:0/165775537 shutdown_connections 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 --2- 172.21.15.156:0/165775537 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806c650 0x7f4e181103e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 --2- 172.21.15.156:0/165775537 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.289 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 --2- 172.21.15.156:0/165775537 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806bc90 0x7f4e1806c110 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 -- 172.21.15.156:0/165775537 >> 172.21.15.156:0/165775537 conn(0x7f4e18079330 msgr2=0x7f4e18079740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 -- 172.21.15.156:0/165775537 shutdown_connections 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 -- 172.21.15.156:0/165775537 wait complete. 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.287+0000 7f4e1df23700 1 Processor -- start 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 -- start start 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806bc90 0x7f4e1819f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18113790 con 0x7f4e1806c650 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18113610 con 0x7f4e18109650 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e1df23700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18113910 con 0x7f4e1806bc90 2024-05-26T12:00:04.290 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:56556/0 (socket says 172.21.15.156:56556) 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 -- 172.21.15.156:0/1102319945 learned_addr learned my addr 172.21.15.156:0/1102319945 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e177fe700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806bc90 0x7f4e1819f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e17fff700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 -- 172.21.15.156:0/1102319945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806bc90 msgr2=0x7f4e1819f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806bc90 0x7f4e1819f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 -- 172.21.15.156:0/1102319945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 msgr2=0x7f4e18199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e16ffd700 1 -- 172.21.15.156:0/1102319945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1819a170 con 0x7f4e1806c650 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.288+0000 7f4e17fff700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e16ffd700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00b580 tx=0x7f4e0c0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c027070 con 0x7f4e1806c650 2024-05-26T12:00:04.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e1819a400 con 0x7f4e1806c650 2024-05-26T12:00:04.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e0c003fa0 con 0x7f4e1806c650 2024-05-26T12:00:04.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c022d10 con 0x7f4e1806c650 2024-05-26T12:00:04.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.289+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e181a6720 con 0x7f4e1806c650 2024-05-26T12:00:04.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.290+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e18104c80 con 0x7f4e1806c650 2024-05-26T12:00:04.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.291+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f4e0c02a070 con 0x7f4e1806c650 2024-05-26T12:00:04.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.292+0000 7f4e14ff9700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 0x7f4e000b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:04.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.292+0000 7f4e177fe700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 0x7f4e000b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:04.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.292+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4e0c0e30d0 con 0x7f4e1806c650 2024-05-26T12:00:04.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.292+0000 7f4e177fe700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 0x7f4e000b4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009550 tx=0x7f4e080091f0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:04.296 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.295+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4e0c031020 con 0x7f4e1806c650 2024-05-26T12:00:04.305 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 podman[67204]: 2024-05-26 12:00:04.053765179 +0000 UTC m=+0.394124962 container start b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-26T12:00:04.305 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mds[67223]: -- 172.21.15.186:0/3272349530 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5564c99b9a40 con 0x5564c998c800 2024-05-26T12:00:04.305 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: starting mds.h at 2024-05-26T12:00:04.305 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 bash[67204]: b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed 2024-05-26T12:00:04.305 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:00:04 smithi186 systemd[1]: Started Ceph mds.h for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:04.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:04.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:04.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:04.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:04 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:04.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.561+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e1802e560 con 0x7f4e1806c650 2024-05-26T12:00:04.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.561+0000 7f4e14ff9700 1 -- 172.21.15.156:0/1102319945 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f4e0c013070 con 0x7f4e1806c650 2024-05-26T12:00:04.562 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:04.562 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 msgr2=0x7f4e000b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 0x7f4e000b4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4e08009550 tx=0x7f4e080091f0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 msgr2=0x7f4e1819fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00b580 tx=0x7f4e0c0057c0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 shutdown_connections 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4e000b2880 0x7f4e000b4d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e18109650 0x7f4e18199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4e1806bc90 0x7f4e1819f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 --2- 172.21.15.156:0/1102319945 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4e1806c650 0x7f4e1819fd90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 >> 172.21.15.156:0/1102319945 conn(0x7f4e18079330 msgr2=0x7f4e181013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:04.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 shutdown_connections 2024-05-26T12:00:04.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:04.564+0000 7f4e1df23700 1 -- 172.21.15.156:0/1102319945 wait complete. 2024-05-26T12:00:04.566 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 9 2024-05-26T12:00:04.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:04 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:04.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:04.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:04.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:04 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:05.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:05 smithi156 ceph-mon[32025]: Deploying daemon mds.i on smithi192 2024-05-26T12:00:05.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:05 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:boot 2024-05-26T12:00:05.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:05 smithi156 ceph-mon[32025]: fsmap 8 up:standby 2024-05-26T12:00:05.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:05 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:00:05.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:05 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1102319945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:05.573 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:05 smithi192 ceph-mon[36545]: Deploying daemon mds.i on smithi192 2024-05-26T12:00:05.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:05 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:boot 2024-05-26T12:00:05.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:05 smithi192 ceph-mon[36545]: fsmap 8 up:standby 2024-05-26T12:00:05.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:05 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:00:05.574 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:05 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1102319945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:05.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:05 smithi186 ceph-mon[36412]: Deploying daemon mds.i on smithi192 2024-05-26T12:00:05.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:05 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:boot 2024-05-26T12:00:05.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:05 smithi186 ceph-mon[36412]: fsmap 8 up:standby 2024-05-26T12:00:05.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:05 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:00:05.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:05 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1102319945' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:06.541 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:06 smithi192 ceph-mon[36545]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:06.541 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 systemd[1]: Starting Ceph mds.i for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:00:06.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:06 smithi186 ceph-mon[36412]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:06.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:06 smithi156 ceph-mon[32025]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:06.812 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 podman[67096]: 2024-05-26 12:00:06.443786424 +0000 UTC m=+0.016813165 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:00:06.812 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 podman[67096]: 2024-05-26 12:00:06.568580874 +0000 UTC m=+0.141607660 container create 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:00:06.812 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 podman[67096]: 2024-05-26 12:00:06.812227074 +0000 UTC m=+0.385253815 container init 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True) 2024-05-26T12:00:07.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 podman[67096]: 2024-05-26 12:00:06.814843368 +0000 UTC m=+0.387870120 container start 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-05-26T12:00:07.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 ceph-mds[67115]: -- 172.21.15.192:0/2815230181 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fe845f1a40 con 0x55fe845c4c00 2024-05-26T12:00:07.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: starting mds.i at 2024-05-26T12:00:07.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 bash[67096]: 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 2024-05-26T12:00:07.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:00:06 smithi192 systemd[1]: Started Ceph mds.i for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:07.267 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:00:07.518 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:08.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:08.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: Deploying daemon mds.g on smithi156 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:boot 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: fsmap 9 up:standby 2024-05-26T12:00:08.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: Deploying daemon mds.g on smithi156 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:boot 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: fsmap 9 up:standby 2024-05-26T12:00:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: Deploying daemon mds.g on smithi156 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:boot 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: fsmap 9 up:standby 2024-05-26T12:00:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:00:09.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 -- 172.21.15.156:0/3220536098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 msgr2=0x7fb4a406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 --2- 172.21.15.156:0/3220536098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a406b6a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009990 tx=0x7fb4a0009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 -- 172.21.15.156:0/3220536098 shutdown_connections 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 --2- 172.21.15.156:0/3220536098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a406bbe0 0x7fb4a41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 --2- 172.21.15.156:0/3220536098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a41095b0 0x7fb4a4109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 --2- 172.21.15.156:0/3220536098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a406b6a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 -- 172.21.15.156:0/3220536098 >> 172.21.15.156:0/3220536098 conn(0x7fb4a40793f0 msgr2=0x7fb4a4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 -- 172.21.15.156:0/3220536098 shutdown_connections 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.757+0000 7fb4abce8700 1 -- 172.21.15.156:0/3220536098 wait complete. 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 Processor -- start 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 -- start start 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a406bbe0 0x7fb4a419fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:09.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a41095b0 0x7fb4a4199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a41136e0 con 0x7fb4a406b220 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4113560 con 0x7fb4a406bbe0 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4abce8700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb4a4113860 con 0x7fb4a41095b0 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4a9a84700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4a9a84700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:34052/0 (socket says 172.21.15.156:34052) 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.758+0000 7fb4a9a84700 1 -- 172.21.15.156:0/4037999302 learned_addr learned my addr 172.21.15.156:0/4037999302 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9283700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a406bbe0 0x7fb4a419fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4aa285700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a41095b0 0x7fb4a4199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 -- 172.21.15.156:0/4037999302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a41095b0 msgr2=0x7fb4a4199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a41095b0 0x7fb4a4199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 -- 172.21.15.156:0/4037999302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a406bbe0 msgr2=0x7fb4a419fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:09.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a406bbe0 0x7fb4a419fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 -- 172.21.15.156:0/4037999302 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4a419a160 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4a9a84700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb49400c9b0 tx=0x7fb49400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494007b00 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4a419a450 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb49400ce50 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb494018660 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.759+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4a41a6710 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.760+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4a4104c70 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.761+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fb49400f450 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.762+0000 7fb49affd700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 0x7fb4900b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.762+0000 7fb4a9283700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 0x7fb4900b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.762+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb4940e1eb0 con 0x7fb4a406b220 2024-05-26T12:00:09.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.762+0000 7fb4a9283700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 0x7fb4900b4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000b5c0 tx=0x7fb4a0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:09.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:09.765+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb494014030 con 0x7fb4a406b220 2024-05-26T12:00:10.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.029+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb4a419af30 con 0x7fb4a406b220 2024-05-26T12:00:10.030 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:09 smithi156 ceph-mon[32025]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:10.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.030+0000 7fb49affd700 1 -- 172.21.15.156:0/4037999302 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8013 (secure 0 0 0) 0x7fb4940aa100 con 0x7fb4a406b220 2024-05-26T12:00:10.031 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:10.031 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 msgr2=0x7fb4900b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 0x7fb4900b4d40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb4a000b5c0 tx=0x7fb4a0005960 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 msgr2=0x7fb4a419f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb49400c9b0 tx=0x7fb49400ccc0 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 shutdown_connections 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4900b2880 0x7fb4900b4d40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb4a41095b0 0x7fb4a4199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb4a406b220 0x7fb4a419f840 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 --2- 172.21.15.156:0/4037999302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a406bbe0 0x7fb4a419fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 >> 172.21.15.156:0/4037999302 conn(0x7fb4a40793f0 msgr2=0x7fb4a4101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:10.033 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 shutdown_connections 2024-05-26T12:00:10.034 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:10.032+0000 7fb4abce8700 1 -- 172.21.15.156:0/4037999302 wait complete. 2024-05-26T12:00:10.034 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 10 2024-05-26T12:00:10.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:09 smithi186 ceph-mon[36412]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:10.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:09 smithi192 ceph-mon[36545]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:10.758 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:10 smithi156 systemd[1]: Starting Ceph mds.g for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:00:11.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:10 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/4037999302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:11.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:10 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/4037999302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:11.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:10 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/4037999302' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:11.680 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 podman[70565]: 2024-05-26 12:00:11.309677583 +0000 UTC m=+0.017266780 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:00:11.680 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 podman[70565]: 2024-05-26 12:00:11.443350015 +0000 UTC m=+0.150939191 container create 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-26T12:00:11.680 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 podman[70565]: 2024-05-26 12:00:11.680012676 +0000 UTC m=+0.387601871 container init 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-26T12:00:11.979 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 podman[70565]: 2024-05-26 12:00:11.683106073 +0000 UTC m=+0.390695275 container start 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-26T12:00:11.979 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mds[70584]: -- 172.21.15.156:0/4242229716 <== mon.0 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584ebe8fa40 con 0x5584ebe62c00 2024-05-26T12:00:11.979 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: starting mds.g at 2024-05-26T12:00:11.979 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 bash[70565]: 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 2024-05-26T12:00:11.979 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:00:11 smithi156 systemd[1]: Started Ceph mds.g for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:12.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:11 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:11 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:12.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:12.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:12.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:12.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:12.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:11 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:13.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:12 smithi156 ceph-mon[32025]: Deploying daemon mds.a on smithi156 2024-05-26T12:00:13.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:12 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6838/2826647780,v1:172.21.15.156:6839/2826647780] up:boot 2024-05-26T12:00:13.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:12 smithi156 ceph-mon[32025]: fsmap 10 up:standby 2024-05-26T12:00:13.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:12 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:00:13.178 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:00:13.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:12 smithi192 ceph-mon[36545]: Deploying daemon mds.a on smithi156 2024-05-26T12:00:13.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:12 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6838/2826647780,v1:172.21.15.156:6839/2826647780] up:boot 2024-05-26T12:00:13.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:12 smithi192 ceph-mon[36545]: fsmap 10 up:standby 2024-05-26T12:00:13.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:12 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:00:13.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:12 smithi186 ceph-mon[36412]: Deploying daemon mds.a on smithi156 2024-05-26T12:00:13.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:12 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6838/2826647780,v1:172.21.15.156:6839/2826647780] up:boot 2024-05-26T12:00:13.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:12 smithi186 ceph-mon[36412]: fsmap 10 up:standby 2024-05-26T12:00:13.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:12 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:00:13.600 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:14.120 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:13 smithi156 ceph-mon[32025]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:14.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:13 smithi192 ceph-mon[36545]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:14.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:13 smithi186 ceph-mon[36412]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 -- 172.21.15.156:0/2675119559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b41095b0 msgr2=0x7f17b4109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 --2- 172.21.15.156:0/2675119559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b41095b0 0x7f17b4109990 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f17b0009990 tx=0x7f17b0009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 -- 172.21.15.156:0/2675119559 shutdown_connections 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 --2- 172.21.15.156:0/2675119559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b406bbe0 0x7f17b41103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 --2- 172.21.15.156:0/2675119559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b41095b0 0x7f17b4109990 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 --2- 172.21.15.156:0/2675119559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 0x7f17b406b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 -- 172.21.15.156:0/2675119559 >> 172.21.15.156:0/2675119559 conn(0x7f17b40793f0 msgr2=0x7f17b4079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 -- 172.21.15.156:0/2675119559 shutdown_connections 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.456+0000 7f17bc61f700 1 -- 172.21.15.156:0/2675119559 wait complete. 2024-05-26T12:00:15.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 Processor -- start 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 -- start start 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 0x7f17b4199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b41095b0 0x7f17b419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f17b4113740 con 0x7f17b406bbe0 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f17b41135c0 con 0x7f17b41095b0 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17bc61f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f17b41138c0 con 0x7f17b406b220 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43654/0 (socket says 172.21.15.156:43654) 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 -- 172.21.15.156:0/46416224 learned_addr learned my addr 172.21.15.156:0/46416224 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17ba3bb700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 0x7f17b4199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17babbc700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b41095b0 0x7f17b419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:15.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 -- 172.21.15.156:0/46416224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 msgr2=0x7f17b4199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 0x7f17b4199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 -- 172.21.15.156:0/46416224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b41095b0 msgr2=0x7f17b419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b41095b0 0x7f17b419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.457+0000 7f17b9bba700 1 -- 172.21.15.156:0/46416224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b419ee10 con 0x7f17b406bbe0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17b9bba700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f17a400b530 tx=0x7f17a400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4015020 con 0x7f17b406bbe0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17a4011ce0 con 0x7f17b406bbe0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a4010e40 con 0x7f17b406bbe0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b419f100 con 0x7f17b406bbe0 2024-05-26T12:00:15.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.458+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b41a6980 con 0x7f17b406bbe0 2024-05-26T12:00:15.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.459+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b410ba90 con 0x7f17b406bbe0 2024-05-26T12:00:15.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.460+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f17a4018020 con 0x7f17b406bbe0 2024-05-26T12:00:15.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.461+0000 7f17ab7fe700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 0x7f17a00b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:15.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.461+0000 7f17ba3bb700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 0x7f17a00b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:15.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.461+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f17a40e15c0 con 0x7f17b406bbe0 2024-05-26T12:00:15.463 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.461+0000 7f17ba3bb700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 0x7f17a00b4dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f17b000b5c0 tx=0x7f17b0005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:15.465 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.465+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f17a401f020 con 0x7f17b406bbe0 2024-05-26T12:00:15.731 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.731+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f17b419b160 con 0x7f17b406bbe0 2024-05-26T12:00:15.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.731+0000 7f17ab7fe700 1 -- 172.21.15.156:0/46416224 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8842 (secure 0 0 0) 0x7f17a40a9780 con 0x7f17b406bbe0 2024-05-26T12:00:15.732 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:15.732 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-26T12:00:15.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 msgr2=0x7f17a00b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:15.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 0x7f17a00b4dc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f17b000b5c0 tx=0x7f17b0005960 comp rx=0 tx=0).stop 2024-05-26T12:00:15.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 msgr2=0x7f17b419a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:15.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f17a400b530 tx=0x7f17a400b840 comp rx=0 tx=0).stop 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 shutdown_connections 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f17a00b2900 0x7f17a00b4dc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17b41095b0 0x7f17b419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f17b406b220 0x7f17b4199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 --2- 172.21.15.156:0/46416224 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f17b406bbe0 0x7f17b419a340 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 >> 172.21.15.156:0/46416224 conn(0x7f17b40793f0 msgr2=0x7f17b4101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 shutdown_connections 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:15.734+0000 7f17bc61f700 1 -- 172.21.15.156:0/46416224 wait complete. 2024-05-26T12:00:15.735 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 11 2024-05-26T12:00:16.278 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:15 smithi186 ceph-mon[36412]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:16.279 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:15 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/46416224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:16.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:15 smithi156 ceph-mon[32025]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:16.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:15 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/46416224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:16.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:15 smithi192 ceph-mon[36545]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:16.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:15 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/46416224' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:16.794 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:00:16 smithi156 systemd[1]: Started Ceph mds.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:17.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:17.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6840/1148384159,v1:172.21.15.156:6841/1148384159] up:boot 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: fsmap 11 up:standby 2024-05-26T12:00:17.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:17 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6840/1148384159,v1:172.21.15.156:6841/1148384159] up:boot 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: fsmap 11 up:standby 2024-05-26T12:00:18.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:17 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6840/1148384159,v1:172.21.15.156:6841/1148384159] up:boot 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: fsmap 11 up:standby 2024-05-26T12:00:18.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:17 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:00:18.412 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:00:18.670 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:18.792 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:18 smithi156 ceph-mon[32025]: Deploying daemon mds.b on smithi186 2024-05-26T12:00:18.793 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:18 smithi156 ceph-mon[32025]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:18.929 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:18 smithi186 ceph-mon[36412]: Deploying daemon mds.b on smithi186 2024-05-26T12:00:18.930 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:18 smithi186 ceph-mon[36412]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:19.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:18 smithi192 ceph-mon[36545]: Deploying daemon mds.b on smithi186 2024-05-26T12:00:19.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:18 smithi192 ceph-mon[36545]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:19.227 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 podman[67520]: 2024-05-26 12:00:19.16281993 +0000 UTC m=+0.130147284 container create 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:00:19.726 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 podman[67520]: 2024-05-26 12:00:19.390513972 +0000 UTC m=+0.357841285 container init 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-26T12:00:19.727 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 podman[67520]: 2024-05-26 12:00:19.393049904 +0000 UTC m=+0.360377216 container start 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-26T12:00:19.727 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 ceph-mds[67538]: -- 172.21.15.186:0/2139779017 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556ed099a40 con 0x5556ed06c800 2024-05-26T12:00:19.727 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: starting mds.b at 2024-05-26T12:00:19.727 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 bash[67520]: 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 2024-05-26T12:00:19.727 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:19 smithi186 systemd[1]: Started Ceph mds.b for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:00:20.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 -- 172.21.15.156:0/2134039236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f877806bbe0 msgr2=0x7f87781103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 --2- 172.21.15.156:0/2134039236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f877806bbe0 0x7f87781103d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8760009990 tx=0x7f8760009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 -- 172.21.15.156:0/2134039236 shutdown_connections 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 --2- 172.21.15.156:0/2134039236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f877806bbe0 0x7f87781103d0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 --2- 172.21.15.156:0/2134039236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f87781095b0 0x7f8778109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 --2- 172.21.15.156:0/2134039236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806b220 0x7f877806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 -- 172.21.15.156:0/2134039236 >> 172.21.15.156:0/2134039236 conn(0x7f87780793f0 msgr2=0x7f8778079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 -- 172.21.15.156:0/2134039236 shutdown_connections 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.332+0000 7f877efdf700 1 -- 172.21.15.156:0/2134039236 wait complete. 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 Processor -- start 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 -- start start 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 0x7f8778102510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806bbe0 0x7f8778102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:20.362 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f87781136e0 con 0x7f87781095b0 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8778113560 con 0x7f877806bbe0 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877efdf700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8778113860 con 0x7f877806b220 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43684/0 (socket says 172.21.15.156:43684) 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 -- 172.21.15.156:0/3357645581 learned_addr learned my addr 172.21.15.156:0/3357645581 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877cd7b700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 0x7f8778102510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f8777fff700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806bbe0 0x7f8778102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 -- 172.21.15.156:0/3357645581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 msgr2=0x7f8778102510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 0x7f8778102510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 -- 172.21.15.156:0/3357645581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806bbe0 msgr2=0x7f8778102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:20.363 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806bbe0 0x7f8778102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877d57c700 1 -- 172.21.15.156:0/3357645581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87781a8140 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.333+0000 7f877cd7b700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 0x7f8778102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f877d57c700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f876000c010 tx=0x7f87600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760027070 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8760022470 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87781a83d0 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8760017650 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.334+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87781a8820 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.335+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877810ba90 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.336+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f876002a070 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.337+0000 7f8775ffb700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 0x7f87640b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.337+0000 7f877cd7b700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 0x7f87640b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.337+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f87600e2ff0 con 0x7f87781095b0 2024-05-26T12:00:20.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.338+0000 7f877cd7b700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 0x7f87640b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8770009fd0 tx=0x7f8770009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:20.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.341+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8760013070 con 0x7f87781095b0 2024-05-26T12:00:20.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.613+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f877806a230 con 0x7f87781095b0 2024-05-26T12:00:20.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:20.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:boot 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: fsmap 12 up:standby 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:00:20.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:20 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:20.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.614+0000 7f8775ffb700 1 -- 172.21.15.156:0/3357645581 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10500 (secure 0 0 0) 0x7f87600ab1a0 con 0x7f87781095b0 2024-05-26T12:00:20.616 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:20.616 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14601,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14613,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24397,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24418,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24427,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24442,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24449,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24458,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24476,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 msgr2=0x7f87640b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 0x7f87640b4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8770009fd0 tx=0x7f8770009260 comp rx=0 tx=0).stop 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 msgr2=0x7f8778102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f876000c010 tx=0x7f87600047b0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 shutdown_connections 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f87640b2880 0x7f87640b4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f87781095b0 0x7f8778102f90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f877806b220 0x7f8778102510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 --2- 172.21.15.156:0/3357645581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f877806bbe0 0x7f8778102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:20.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 >> 172.21.15.156:0/3357645581 conn(0x7f87780793f0 msgr2=0x7f8778101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:20.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 shutdown_connections 2024-05-26T12:00:20.618 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:20.616+0000 7f877efdf700 1 -- 172.21.15.156:0/3357645581 wait complete. 2024-05-26T12:00:20.618 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 13 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:boot 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: fsmap 12 up:standby 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:00:20.816 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:20 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:20.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:20.823 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:boot 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: fsmap 12 up:standby 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:00:20.824 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:20 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:21.530 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph --format=json-pretty osd lspools 2024-05-26T12:00:21.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:21 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3357645581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:21.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:21 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:21 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3357645581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:21.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:21 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.963 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:21 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3357645581' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:00:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:21.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:21 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:22 smithi156 ceph-mon[32025]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:22.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:22 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:22 smithi192 ceph-mon[36545]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:22 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:22 smithi186 ceph-mon[36412]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:22.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:22.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:22 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:23.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- 172.21.15.156:0/131141239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868102e00 msgr2=0x7f286810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:23.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 --2- 172.21.15.156:0/131141239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868102e00 0x7f286810f2f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f286000b0a0 tx=0x7f286000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.696 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- 172.21.15.156:0/131141239 shutdown_connections 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 --2- 172.21.15.156:0/131141239 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868102e00 0x7f286810f2f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 --2- 172.21.15.156:0/131141239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868108520 0x7f2868108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 --2- 172.21.15.156:0/131141239 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102440 0x7f28681028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- 172.21.15.156:0/131141239 >> 172.21.15.156:0/131141239 conn(0x7f28680793f0 msgr2=0x7f2868079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- 172.21.15.156:0/131141239 shutdown_connections 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- 172.21.15.156:0/131141239 wait complete. 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 Processor -- start 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.694+0000 7f286f811700 1 -- start start 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868102440 0x7f286819f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102e00 0x7f286819fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f28681125f0 con 0x7f2868108520 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2868112470 con 0x7f2868102440 2024-05-26T12:00:23.697 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286f811700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2868112770 con 0x7f2868102e00 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43710/0 (socket says 172.21.15.156:43710) 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 -- 172.21.15.156:0/1366744956 learned_addr learned my addr 172.21.15.156:0/1366744956 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286d5ad700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868102440 0x7f286819f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286cdac700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102e00 0x7f286819fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 -- 172.21.15.156:0/1366744956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102e00 msgr2=0x7f286819fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102e00 0x7f286819fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 -- 172.21.15.156:0/1366744956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868102440 msgr2=0x7f286819f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868102440 0x7f286819f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 -- 172.21.15.156:0/1366744956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286819a190 con 0x7f2868108520 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.695+0000 7f286ddae700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f286000b830 tx=0x7f2860004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.696+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860021070 con 0x7f2868108520 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.696+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286819a420 con 0x7f2868108520 2024-05-26T12:00:23.698 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.696+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2868103fb0 con 0x7f2868108520 2024-05-26T12:00:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.696+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2860007e30 con 0x7f2868108520 2024-05-26T12:00:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.696+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2860004700 con 0x7f2868108520 2024-05-26T12:00:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.698+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f284c005330 con 0x7f2868108520 2024-05-26T12:00:23.699 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.699+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f2860003c00 con 0x7f2868108520 2024-05-26T12:00:23.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.699+0000 7f285e7fc700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 0x7f28540b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:23.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.700+0000 7f286d5ad700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 0x7f28540b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:23.700 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.700+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f286000f020 con 0x7f2868108520 2024-05-26T12:00:23.701 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.700+0000 7f286d5ad700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 0x7f28540b4dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28681078b0 tx=0x7f286400b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:23.706 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.706+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f286002a020 con 0x7f2868108520 2024-05-26T12:00:23.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.971+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f284c0058f0 con 0x7f2868108520 2024-05-26T12:00:23.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.972+0000 7f285e7fc700 1 -- 172.21.15.156:0/1366744956 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f28600ab340 con 0x7f2868108520 2024-05-26T12:00:23.972 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout:[ 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout: { 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout: "poolnum": 1, 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout: "poolname": ".mgr" 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout: } 2024-05-26T12:00:23.973 INFO:teuthology.orchestra.run.smithi156.stdout:] 2024-05-26T12:00:23.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.974+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 msgr2=0x7f28540b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:23.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.974+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 0x7f28540b4dc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f28681078b0 tx=0x7f286400b410 comp rx=0 tx=0).stop 2024-05-26T12:00:23.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.974+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 msgr2=0x7f2868199a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:23.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.974+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f286000b830 tx=0x7f2860004ce0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.975 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 shutdown_connections 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f28540b2900 0x7f28540b4dc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2868108520 0x7f2868199a30 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2868102440 0x7f286819f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 --2- 172.21.15.156:0/1366744956 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2868102e00 0x7f286819fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 >> 172.21.15.156:0/1366744956 conn(0x7f28680793f0 msgr2=0x7f286806b5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 shutdown_connections 2024-05-26T12:00:23.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:23.975+0000 7f286f811700 1 -- 172.21.15.156:0/1366744956 wait complete. 2024-05-26T12:00:24.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:24 smithi156 ceph-mon[32025]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:24.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:24 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.614 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:24 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1366744956' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-26T12:00:24.640 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-26T12:00:24.641 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-26T12:00:24.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:24 smithi192 ceph-mon[36545]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:24.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:24 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:24 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1366744956' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-26T12:00:24.890 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:24 smithi186 ceph-mon[36412]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:24 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:24 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1366744956' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-26T12:00:25.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:25.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:25.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:25.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:25 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:25.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:25.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:25.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:25.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:25 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:25.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:25.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:25.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:25.889 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:25 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 -- 172.21.15.156:0/681831816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102df0 msgr2=0x7fbd6010f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/681831816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102df0 0x7fbd6010f2e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b0a0 tx=0x7fbd5800b3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 -- 172.21.15.156:0/681831816 shutdown_connections 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/681831816 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102df0 0x7fbd6010f2e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/681831816 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/681831816 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102430 0x7fbd601028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 -- 172.21.15.156:0/681831816 >> 172.21.15.156:0/681831816 conn(0x7fbd600793b0 msgr2=0x7fbd600797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:26.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 -- 172.21.15.156:0/681831816 shutdown_connections 2024-05-26T12:00:26.399 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.397+0000 7fbd67d6b700 1 -- 172.21.15.156:0/681831816 wait complete. 2024-05-26T12:00:26.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 Processor -- start 2024-05-26T12:00:26.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 -- start start 2024-05-26T12:00:26.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102430 0x7fbd6019aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:26.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 0x7fbd6019af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:26.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbd601125e0 con 0x7fbd60102430 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbd60112460 con 0x7fbd60102df0 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd67d6b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbd60112760 con 0x7fbd60108510 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd65306700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 0x7fbd6019af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.398+0000 7fbd65b07700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102430 0x7fbd6019aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.156:42050/0 (socket says 172.21.15.156:42050) 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 -- 172.21.15.156:0/978074421 learned_addr learned my addr 172.21.15.156:0/978074421 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd65306700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 0x7fbd6019af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.156:48508/0 (socket says 172.21.15.156:48508) 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 -- 172.21.15.156:0/978074421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 msgr2=0x7fbd6019af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 0x7fbd6019af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 -- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102430 msgr2=0x7fbd6019aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102430 0x7fbd6019aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:26.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 -- 172.21.15.156:0/978074421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd601a5fc0 con 0x7fbd60108510 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd66308700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b780 tx=0x7fbd58004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58021070 con 0x7fbd60108510 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd601987f0 con 0x7fbd60108510 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbd58007e30 con 0x7fbd60108510 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58004700 con 0x7fbd60108510 2024-05-26T12:00:26.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.399+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd60198d80 con 0x7fbd60108510 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.401+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd6010a9a0 con 0x7fbd60108510 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.402+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fbd58007980 con 0x7fbd60108510 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.402+0000 7fbd56ffd700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 0x7fbd4c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.403+0000 7fbd65b07700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 0x7fbd4c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.403+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fbd5800f020 con 0x7fbd60108510 2024-05-26T12:00:26.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.403+0000 7fbd65b07700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 0x7fbd4c0b4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009be0 tx=0x7fbd5c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:26.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.407+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbd5802a020 con 0x7fbd60108510 2024-05-26T12:00:26.637 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:26.636+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fbd6006c070 con 0x7fbd60108510 2024-05-26T12:00:26.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:26 smithi192 ceph-mon[36545]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:26.912 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:26 smithi156 ceph-mon[32025]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:26.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:26 smithi186 ceph-mon[36412]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:27.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.565+0000 7fbd56ffd700 1 -- 172.21.15.156:0/978074421 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7fbd580ab320 con 0x7fbd60108510 2024-05-26T12:00:27.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 msgr2=0x7fbd4c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:27.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 0x7fbd4c0b4d40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c009be0 tx=0x7fbd5c009260 comp rx=0 tx=0).stop 2024-05-26T12:00:27.570 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 msgr2=0x7fbd601a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800b780 tx=0x7fbd58004d00 comp rx=0 tx=0).stop 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 shutdown_connections 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbd4c0b2880 0x7fbd4c0b4d40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbd60108510 0x7fbd601a5a80 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbd60102430 0x7fbd6019aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 --2- 172.21.15.156:0/978074421 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbd60102df0 0x7fbd6019af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 >> 172.21.15.156:0/978074421 conn(0x7fbd600793b0 msgr2=0x7fbd60100480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 shutdown_connections 2024-05-26T12:00:27.571 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:27.569+0000 7fbd67d6b700 1 -- 172.21.15.156:0/978074421 wait complete. 2024-05-26T12:00:27.572 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' created 2024-05-26T12:00:27.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/978074421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:27 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/978074421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.893 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:27 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/978074421' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:27 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.370 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-26T12:00:28.618 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:28.740 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: osdmap e73: 12 total, 12 up, 12 in 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:28.741 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-26T12:00:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: osdmap e73: 12 total, 12 up, 12 in 2024-05-26T12:00:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:28.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:28.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:28.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: osdmap e73: 12 total, 12 up, 12 in 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:00:29.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:00:29.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:00:29.904 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:29 smithi156 ceph-mon[32025]: osdmap e74: 12 total, 12 up, 12 in 2024-05-26T12:00:29.905 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:29 smithi156 ceph-mon[32025]: osdmap e75: 12 total, 12 up, 12 in 2024-05-26T12:00:29.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:29 smithi186 ceph-mon[36412]: osdmap e74: 12 total, 12 up, 12 in 2024-05-26T12:00:29.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:29 smithi186 ceph-mon[36412]: osdmap e75: 12 total, 12 up, 12 in 2024-05-26T12:00:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:29 smithi192 ceph-mon[36545]: osdmap e74: 12 total, 12 up, 12 in 2024-05-26T12:00:30.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:29 smithi192 ceph-mon[36545]: osdmap e75: 12 total, 12 up, 12 in 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.374+0000 7f5901aa5700 1 -- 172.21.15.156:0/1987375771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc1095b0 msgr2=0x7f58fc109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.374+0000 7f5901aa5700 1 --2- 172.21.15.156:0/1987375771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc1095b0 0x7f58fc109990 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f58ec009990 tx=0x7f58ec009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 -- 172.21.15.156:0/1987375771 shutdown_connections 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 --2- 172.21.15.156:0/1987375771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06bbe0 0x7f58fc1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 --2- 172.21.15.156:0/1987375771 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc1095b0 0x7f58fc109990 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 --2- 172.21.15.156:0/1987375771 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc06b220 0x7f58fc06b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 -- 172.21.15.156:0/1987375771 >> 172.21.15.156:0/1987375771 conn(0x7f58fc0793f0 msgr2=0x7f58fc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:30.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 -- 172.21.15.156:0/1987375771 shutdown_connections 2024-05-26T12:00:30.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 -- 172.21.15.156:0/1987375771 wait complete. 2024-05-26T12:00:30.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.375+0000 7f5901aa5700 1 Processor -- start 2024-05-26T12:00:30.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 -- start start 2024-05-26T12:00:30.378 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06b220 0x7f58fc19f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 0x7f58fc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1136c0 con 0x7f58fc06bbe0 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc113540 con 0x7f58fc06b220 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f5901aa5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc113840 con 0x7f58fc1095b0 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55076/0 (socket says 172.21.15.156:55076) 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 -- 172.21.15.156:0/2188170796 learned_addr learned my addr 172.21.15.156:0/2188170796 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58fb7fe700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06b220 0x7f58fc19f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:30.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58fbfff700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 0x7f58fc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 -- 172.21.15.156:0/2188170796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 msgr2=0x7f58fc1999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 0x7f58fc1999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 -- 172.21.15.156:0/2188170796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06b220 msgr2=0x7f58fc19f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06b220 0x7f58fc19f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 -- 172.21.15.156:0/2188170796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc19a120 con 0x7f58fc06bbe0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58fbfff700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 0x7f58fc1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.376+0000 7f58faffd700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c9b0 tx=0x7f58f000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.377+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0007b00 con 0x7f58fc06bbe0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.377+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f58f0007ca0 con 0x7f58fc06bbe0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.377+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc19a410 con 0x7f58fc06bbe0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.377+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0018730 con 0x7f58fc06bbe0 2024-05-26T12:00:30.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.377+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc1a6680 con 0x7f58fc06bbe0 2024-05-26T12:00:30.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.378+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58fc10ba90 con 0x7f58fc06bbe0 2024-05-26T12:00:30.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.379+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f58f000f450 con 0x7f58fc06bbe0 2024-05-26T12:00:30.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.380+0000 7f58f8ff9700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 0x7f58e40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:30.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.380+0000 7f58fb7fe700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 0x7f58e40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:30.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.380+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7497+0+0 (secure 0 0 0) 0x7f58f00e2270 con 0x7f58fc06bbe0 2024-05-26T12:00:30.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.380+0000 7f58fb7fe700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 0x7f58e40b4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f58ec006010 tx=0x7f58ec01a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:30.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.384+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f58f0014030 con 0x7f58fc06bbe0 2024-05-26T12:00:30.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:30.590+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f58fc19aea0 con 0x7f58fc06bbe0 2024-05-26T12:00:30.867 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:30 smithi186 ceph-mon[36412]: pgmap v285: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:30.867 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:30 smithi186 ceph-mon[36412]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:31.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:30 smithi156 ceph-mon[32025]: pgmap v285: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:31.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:30 smithi156 ceph-mon[32025]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:31.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:30 smithi192 ceph-mon[36545]: pgmap v285: 33 pgs: 6 creating+peering, 1 active+clean, 26 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:31.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:30 smithi192 ceph-mon[36545]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:31.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.584+0000 7f58f8ff9700 1 -- 172.21.15.156:0/2188170796 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v76) ==== 170+0+0 (secure 0 0 0) 0x7f58f00aa350 con 0x7f58fc06bbe0 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 msgr2=0x7f58e40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 0x7f58e40b4d40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f58ec006010 tx=0x7f58ec01a040 comp rx=0 tx=0).stop 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 msgr2=0x7f58fc19fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f58f000c9b0 tx=0x7f58f000cd70 comp rx=0 tx=0).stop 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 shutdown_connections 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f58e40b2880 0x7f58e40b4d40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f58fc1095b0 0x7f58fc1999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58fc06b220 0x7f58fc19f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:31.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 --2- 172.21.15.156:0/2188170796 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f58fc06bbe0 0x7f58fc19fcf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:31.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 >> 172.21.15.156:0/2188170796 conn(0x7f58fc0793f0 msgr2=0x7f58fc100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:31.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 shutdown_connections 2024-05-26T12:00:31.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:31.588+0000 7f5901aa5700 1 -- 172.21.15.156:0/2188170796 wait complete. 2024-05-26T12:00:31.596 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' created 2024-05-26T12:00:31.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:31 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-26T12:00:31.980 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:31 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-26T12:00:32.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:31 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-26T12:00:32.414 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-26T12:00:32.682 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:32.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:32 smithi156 ceph-mon[32025]: pgmap v287: 33 pgs: 6 creating+peering, 21 active+clean, 6 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:32.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:32 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-26T12:00:32.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:32 smithi156 ceph-mon[32025]: osdmap e76: 12 total, 12 up, 12 in 2024-05-26T12:00:32.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:32 smithi186 ceph-mon[36412]: pgmap v287: 33 pgs: 6 creating+peering, 21 active+clean, 6 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:32.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:32 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-26T12:00:32.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:32 smithi186 ceph-mon[36412]: osdmap e76: 12 total, 12 up, 12 in 2024-05-26T12:00:33.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:32 smithi192 ceph-mon[36545]: pgmap v287: 33 pgs: 6 creating+peering, 21 active+clean, 6 unknown; 577 KiB data, 342 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:33.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:32 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2188170796' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-26T12:00:33.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:32 smithi192 ceph-mon[36545]: osdmap e76: 12 total, 12 up, 12 in 2024-05-26T12:00:33.870 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:33 smithi156 ceph-mon[32025]: osdmap e77: 12 total, 12 up, 12 in 2024-05-26T12:00:33.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:33 smithi186 ceph-mon[36412]: osdmap e77: 12 total, 12 up, 12 in 2024-05-26T12:00:34.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:33 smithi192 ceph-mon[36545]: osdmap e77: 12 total, 12 up, 12 in 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 -- 172.21.15.156:0/2132824577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf18102e90 msgr2=0x7fcf1810f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 --2- 172.21.15.156:0/2132824577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf18102e90 0x7fcf1810f310 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000b0a0 tx=0x7fcf1000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 -- 172.21.15.156:0/2132824577 shutdown_connections 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 --2- 172.21.15.156:0/2132824577 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf18102e90 0x7fcf1810f310 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 --2- 172.21.15.156:0/2132824577 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 0x7fcf181088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 --2- 172.21.15.156:0/2132824577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf181024d0 0x7fcf18102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 -- 172.21.15.156:0/2132824577 >> 172.21.15.156:0/2132824577 conn(0x7fcf18078490 msgr2=0x7fcf180788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 -- 172.21.15.156:0/2132824577 shutdown_connections 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.428+0000 7fcf20019700 1 -- 172.21.15.156:0/2132824577 wait complete. 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 Processor -- start 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 -- start start 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf18102e90 0x7fcf1819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 0x7fcf18198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fcf181125e0 con 0x7fcf181024d0 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18112460 con 0x7fcf18102e90 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf20019700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fcf18112760 con 0x7fcf181084d0 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:34.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55092/0 (socket says 172.21.15.156:55092) 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 -- 172.21.15.156:0/3007192656 learned_addr learned my addr 172.21.15.156:0/3007192656 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1e5b6700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 0x7fcf18198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1d5b4700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf18102e90 0x7fcf1819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 -- 172.21.15.156:0/3007192656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 msgr2=0x7fcf18198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 0x7fcf18198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 -- 172.21.15.156:0/3007192656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf18102e90 msgr2=0x7fcf1819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf18102e90 0x7fcf1819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.429+0000 7fcf1ddb5700 1 -- 172.21.15.156:0/3007192656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf18198fe0 con 0x7fcf181024d0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf1ddb5700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400d980 tx=0x7fcf1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf14016070 con 0x7fcf181024d0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcf14004830 con 0x7fcf181024d0 2024-05-26T12:00:34.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf181992d0 con 0x7fcf181024d0 2024-05-26T12:00:34.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf140054c0 con 0x7fcf181024d0 2024-05-26T12:00:34.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.430+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf181a54e0 con 0x7fcf181024d0 2024-05-26T12:00:34.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.431+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf18103b80 con 0x7fcf181024d0 2024-05-26T12:00:34.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.432+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fcf14004220 con 0x7fcf181024d0 2024-05-26T12:00:34.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.433+0000 7fcf0effd700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 0x7fcf040b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:34.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.433+0000 7fcf1d5b4700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 0x7fcf040b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:34.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.433+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(78..78 src has 1..78) ==== 7879+0+0 (secure 0 0 0) 0x7fcf140e2350 con 0x7fcf181024d0 2024-05-26T12:00:34.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.433+0000 7fcf1d5b4700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 0x7fcf040b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf08005ea0 tx=0x7fcf08005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:34.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.436+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fcf1401f020 con 0x7fcf181024d0 2024-05-26T12:00:34.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:34.688+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fcf1806a230 con 0x7fcf181024d0 2024-05-26T12:00:34.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:34 smithi186 ceph-mon[36412]: pgmap v290: 97 pgs: 22 creating+peering, 27 active+clean, 48 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:34.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:34 smithi186 ceph-mon[36412]: osdmap e78: 12 total, 12 up, 12 in 2024-05-26T12:00:35.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:34 smithi156 ceph-mon[32025]: pgmap v290: 97 pgs: 22 creating+peering, 27 active+clean, 48 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:35.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:34 smithi156 ceph-mon[32025]: osdmap e78: 12 total, 12 up, 12 in 2024-05-26T12:00:35.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:34 smithi192 ceph-mon[36545]: pgmap v290: 97 pgs: 22 creating+peering, 27 active+clean, 48 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:35.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:34 smithi192 ceph-mon[36545]: osdmap e78: 12 total, 12 up, 12 in 2024-05-26T12:00:35.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.611+0000 7fcf0effd700 1 -- 172.21.15.156:0/3007192656 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7fcf140aa220 con 0x7fcf181024d0 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 msgr2=0x7fcf040b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 0x7fcf040b4d40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fcf08005ea0 tx=0x7fcf08005e30 comp rx=0 tx=0).stop 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 msgr2=0x7fcf1819e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400d980 tx=0x7fcf1400dc90 comp rx=0 tx=0).stop 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 shutdown_connections 2024-05-26T12:00:35.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fcf040b2880 0x7fcf040b4d40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fcf181084d0 0x7fcf18198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fcf181024d0 0x7fcf1819e680 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 --2- 172.21.15.156:0/3007192656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf18102e90 0x7fcf1819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.613+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 >> 172.21.15.156:0/3007192656 conn(0x7fcf18078490 msgr2=0x7fcf18100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.614+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 shutdown_connections 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:35.614+0000 7fcf20019700 1 -- 172.21.15.156:0/3007192656 wait complete. 2024-05-26T12:00:35.615 INFO:teuthology.orchestra.run.smithi156.stderr:new fs with metadata pool 2 and data pool 3 2024-05-26T12:00:35.945 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:35 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-26T12:00:35.945 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:35 smithi156 ceph-mon[32025]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-26T12:00:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:35 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-26T12:00:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:35 smithi186 ceph-mon[36412]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-26T12:00:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:35 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-26T12:00:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:35 smithi192 ceph-mon[36545]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-26T12:00:36.404 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:00:36.654 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:36.776 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:36 smithi156 ceph-mon[32025]: pgmap v292: 97 pgs: 16 creating+peering, 65 active+clean, 16 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:36.776 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:36 smithi156 ceph-mon[32025]: osdmap e79: 12 total, 12 up, 12 in 2024-05-26T12:00:36.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:36 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-26T12:00:36.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:36 smithi156 ceph-mon[32025]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-26T12:00:36.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:36 smithi156 ceph-mon[32025]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-26T12:00:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:36 smithi186 ceph-mon[36412]: pgmap v292: 97 pgs: 16 creating+peering, 65 active+clean, 16 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:36 smithi186 ceph-mon[36412]: osdmap e79: 12 total, 12 up, 12 in 2024-05-26T12:00:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:36 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-26T12:00:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:36 smithi186 ceph-mon[36412]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-26T12:00:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:36 smithi186 ceph-mon[36412]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-26T12:00:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:36 smithi192 ceph-mon[36545]: pgmap v292: 97 pgs: 16 creating+peering, 65 active+clean, 16 unknown; 577 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:00:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:36 smithi192 ceph-mon[36545]: osdmap e79: 12 total, 12 up, 12 in 2024-05-26T12:00:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:36 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3007192656' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-26T12:00:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:36 smithi192 ceph-mon[36545]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-26T12:00:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:36 smithi192 ceph-mon[36545]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-26T12:00:37.881 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:37 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:active 2024-05-26T12:00:37.881 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:37 smithi156 ceph-mon[32025]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:37.881 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:37 smithi156 ceph-mon[32025]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:37.881 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:37 smithi156 ceph-mon[32025]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:37.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:37 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:active 2024-05-26T12:00:37.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:37 smithi186 ceph-mon[36412]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:37.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:37 smithi186 ceph-mon[36412]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:37.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:37 smithi186 ceph-mon[36412]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:37 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] up:active 2024-05-26T12:00:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:37 smithi192 ceph-mon[36545]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:37 smithi192 ceph-mon[36545]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:38.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:37 smithi192 ceph-mon[36545]: Health check update: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-26T12:00:38.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 -- 172.21.15.156:0/3845191446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06b220 msgr2=0x7f344c06b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:38.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 --2- 172.21.15.156:0/3845191446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06b220 0x7f344c06b6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f3434009990 tx=0x7f3434009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.379 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 -- 172.21.15.156:0/3845191446 shutdown_connections 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 --2- 172.21.15.156:0/3845191446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c06bbe0 0x7f344c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 --2- 172.21.15.156:0/3845191446 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c1095b0 0x7f344c109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 --2- 172.21.15.156:0/3845191446 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06b220 0x7f344c06b6a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 -- 172.21.15.156:0/3845191446 >> 172.21.15.156:0/3845191446 conn(0x7f344c0793f0 msgr2=0x7f344c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 -- 172.21.15.156:0/3845191446 shutdown_connections 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.376+0000 7f34526c9700 1 -- 172.21.15.156:0/3845191446 wait complete. 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 Processor -- start 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 -- start start 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 0x7f344c19fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c1095b0 0x7f344c199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:38.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1137a0 con 0x7f344c06bbe0 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f344c113620 con 0x7f344c1095b0 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f34526c9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f344c113920 con 0x7f344c06b220 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f3443fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f3443fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55590/0 (socket says 172.21.15.156:55590) 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f3443fff700 1 -- 172.21.15.156:0/2410197954 learned_addr learned my addr 172.21.15.156:0/2410197954 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.377+0000 7f3450c66700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c1095b0 0x7f344c199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f344bfff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 0x7f344c19fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 -- 172.21.15.156:0/2410197954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 msgr2=0x7f344c19fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 0x7f344c19fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 -- 172.21.15.156:0/2410197954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c1095b0 msgr2=0x7f344c199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:38.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c1095b0 0x7f344c199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 -- 172.21.15.156:0/2410197954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c19a400 con 0x7f344c06bbe0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f344bfff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 0x7f344c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3443fff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3434005910 tx=0x7f34340047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434027070 con 0x7f344c06bbe0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3434003f20 con 0x7f344c06bbe0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c19a690 con 0x7f344c06bbe0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434022c70 con 0x7f344c06bbe0 2024-05-26T12:00:38.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.378+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c1a69d0 con 0x7f344c06bbe0 2024-05-26T12:00:38.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.379+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c10ba90 con 0x7f344c06bbe0 2024-05-26T12:00:38.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.381+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f343402a070 con 0x7f344c06bbe0 2024-05-26T12:00:38.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.381+0000 7f3449ffb700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 0x7f34380b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:38.383 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.381+0000 7f344bfff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 0x7f34380b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:38.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.382+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f34340e3620 con 0x7f344c06bbe0 2024-05-26T12:00:38.384 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.382+0000 7f344bfff700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 0x7f34380b4dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f343c0099d0 tx=0x7f343c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:38.385 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.385+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3434031070 con 0x7f344c06bbe0 2024-05-26T12:00:38.596 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.596+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f344c06a230 con 0x7f344c06bbe0 2024-05-26T12:00:38.598 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.597+0000 7f3449ffb700 1 -- 172.21.15.156:0/2410197954 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22265 (secure 0 0 0) 0x7f3434013070 con 0x7f344c06bbe0 2024-05-26T12:00:38.598 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:38.599 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":79,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:00:34.689832+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 msgr2=0x7f34380b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 0x7f34380b4dc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f343c0099d0 tx=0x7f343c008040 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 msgr2=0x7f344c19ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3434005910 tx=0x7f34340047b0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 shutdown_connections 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f34380b2900 0x7f34380b4dc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f344c1095b0 0x7f344c199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f344c06b220 0x7f344c19fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 --2- 172.21.15.156:0/2410197954 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f344c06bbe0 0x7f344c19ff90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:38.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 >> 172.21.15.156:0/2410197954 conn(0x7f344c0793f0 msgr2=0x7f344c1011d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:38.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 shutdown_connections 2024-05-26T12:00:38.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:38.600+0000 7f34526c9700 1 -- 172.21.15.156:0/2410197954 wait complete. 2024-05-26T12:00:38.971 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:38 smithi156 ceph-mon[32025]: pgmap v294: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-26T12:00:38.971 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:38 smithi156 ceph-mon[32025]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-26T12:00:38.971 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:38 smithi156 ceph-mon[32025]: Cluster is now healthy 2024-05-26T12:00:38.971 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:38 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2410197954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:38 smithi186 ceph-mon[36412]: pgmap v294: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-26T12:00:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:38 smithi186 ceph-mon[36412]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-26T12:00:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:38 smithi186 ceph-mon[36412]: Cluster is now healthy 2024-05-26T12:00:38.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:38 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2410197954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:39.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:38 smithi192 ceph-mon[36545]: pgmap v294: 97 pgs: 8 creating+peering, 89 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-26T12:00:39.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:38 smithi192 ceph-mon[36545]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-05-26T12:00:39.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:38 smithi192 ceph-mon[36545]: Cluster is now healthy 2024-05-26T12:00:39.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:38 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2410197954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:39.311 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:00:39.557 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:39.976 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:39 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: 2024-05-26T12:00:39.647+0000 7f2911fe4700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-26T12:00:40.898 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:40 smithi156 ceph-mon[32025]: pgmap v295: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s wr, 3 op/s 2024-05-26T12:00:40.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:40 smithi186 ceph-mon[36412]: pgmap v295: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s wr, 3 op/s 2024-05-26T12:00:41.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:40 smithi192 ceph-mon[36545]: pgmap v295: 97 pgs: 97 active+clean; 577 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 165 B/s wr, 3 op/s 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 -- 172.21.15.156:0/975502322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102440 msgr2=0x7f22801028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 --2- 172.21.15.156:0/975502322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102440 0x7f22801028c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2274009990 tx=0x7f2274009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 -- 172.21.15.156:0/975502322 shutdown_connections 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 --2- 172.21.15.156:0/975502322 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102e00 0x7f228010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 --2- 172.21.15.156:0/975502322 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 0x7f2280108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 --2- 172.21.15.156:0/975502322 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102440 0x7f22801028c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 -- 172.21.15.156:0/975502322 >> 172.21.15.156:0/975502322 conn(0x7f22800793f0 msgr2=0x7f2280079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 -- 172.21.15.156:0/975502322 shutdown_connections 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 -- 172.21.15.156:0/975502322 wait complete. 2024-05-26T12:00:41.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.152+0000 7f2285b5c700 1 Processor -- start 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 -- start start 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 0x7f22801a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22801125f0 con 0x7f2280102e00 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2280112470 con 0x7f2280108520 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f2285b5c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2280112770 con 0x7f2280102440 2024-05-26T12:00:41.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227f7fe700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.192:3300/0 says I am v2:172.21.15.156:37830/0 (socket says 172.21.15.156:37830) 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55596/0 (socket says 172.21.15.156:55596) 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227f7fe700 1 -- 172.21.15.156:0/3877501655 learned_addr learned my addr 172.21.15.156:0/3877501655 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227ffff700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 0x7f22801a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 -- 172.21.15.156:0/3877501655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 msgr2=0x7f228019be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 -- 172.21.15.156:0/3877501655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 msgr2=0x7f22801a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 0x7f22801a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 -- 172.21.15.156:0/3877501655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22801a0e30 con 0x7f2280102e00 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227f7fe700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.153+0000 7f227effd700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2274006010 tx=0x7f22740057a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.154+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274027070 con 0x7f2280102e00 2024-05-26T12:00:41.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.154+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2274003f20 con 0x7f2280102e00 2024-05-26T12:00:41.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.154+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22801a10c0 con 0x7f2280102e00 2024-05-26T12:00:41.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.154+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274022a30 con 0x7f2280102e00 2024-05-26T12:00:41.157 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.154+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22801a89a0 con 0x7f2280102e00 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.155+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2280103b60 con 0x7f2280102e00 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.156+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f227402a070 con 0x7f2280102e00 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.157+0000 7f227cff9700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 0x7f22680b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.157+0000 7f227f7fe700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 0x7f22680b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.157+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f22740e39b0 con 0x7f2280102e00 2024-05-26T12:00:41.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.158+0000 7f227f7fe700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 0x7f22680b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f227000f4d0 tx=0x7f2270005f90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:41.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.161+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2274031020 con 0x7f2280102e00 2024-05-26T12:00:41.372 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.371+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22801a8b70 con 0x7f2280102e00 2024-05-26T12:00:41.373 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.372+0000 7f227cff9700 1 -- 172.21.15.156:0/3877501655 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v79) ==== 74+0+22265 (secure 0 0 0) 0x7f2274013070 con 0x7f2280102e00 2024-05-26T12:00:41.373 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:00:41.373 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":79,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:00:34.689832+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:00:41.375 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 msgr2=0x7f22680b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 0x7f22680b4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f227000f4d0 tx=0x7f2270005f90 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 msgr2=0x7f228019c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f2274006010 tx=0x7f22740057a0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 shutdown_connections 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22680b2880 0x7f22680b4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2280108520 0x7f22801a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2280102440 0x7f228019be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 --2- 172.21.15.156:0/3877501655 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2280102e00 0x7f228019c360 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 >> 172.21.15.156:0/3877501655 conn(0x7f22800793f0 msgr2=0x7f228006b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 shutdown_connections 2024-05-26T12:00:41.376 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:41.375+0000 7f2285b5c700 1 -- 172.21.15.156:0/3877501655 wait complete. 2024-05-26T12:00:41.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:41 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3877501655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:42.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:41 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3877501655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:42.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:41 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3877501655' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:00:42.091 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs set cephfs standby_count_wanted 0 2024-05-26T12:00:42.331 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:42.767 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:42 smithi156 ceph-mon[32025]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-26T12:00:42.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:42 smithi186 ceph-mon[36412]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-26T12:00:43.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:42 smithi192 ceph-mon[36545]: pgmap v296: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 -- 172.21.15.156:0/1834345823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 msgr2=0x7fc2a41088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 --2- 172.21.15.156:0/1834345823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41088a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc2a000b0a0 tx=0x7fc2a000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 -- 172.21.15.156:0/1834345823 shutdown_connections 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 --2- 172.21.15.156:0/1834345823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4103600 0x7fc2a410f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 --2- 172.21.15.156:0/1834345823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41088a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 --2- 172.21.15.156:0/1834345823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4102c40 0x7fc2a41030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 -- 172.21.15.156:0/1834345823 >> 172.21.15.156:0/1834345823 conn(0x7fc2a4078ab0 msgr2=0x7fc2a40ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:44.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 -- 172.21.15.156:0/1834345823 shutdown_connections 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.056+0000 7fc2abc71700 1 -- 172.21.15.156:0/1834345823 wait complete. 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 Processor -- start 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 -- start start 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4102c40 0x7fc2a419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4103600 0x7fc2a419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a41125d0 con 0x7fc2a41084c0 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a4112450 con 0x7fc2a4102c40 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2abc71700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc2a4112750 con 0x7fc2a4103600 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55622/0 (socket says 172.21.15.156:55622) 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 -- 172.21.15.156:0/630006961 learned_addr learned my addr 172.21.15.156:0/630006961 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:44.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2a9a0d700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4102c40 0x7fc2a419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2a920c700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4103600 0x7fc2a419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 -- 172.21.15.156:0/630006961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4103600 msgr2=0x7fc2a419fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4103600 0x7fc2a419fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 -- 172.21.15.156:0/630006961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4102c40 msgr2=0x7fc2a419f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4102c40 0x7fc2a419f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.057+0000 7fc2aa20e700 1 -- 172.21.15.156:0/630006961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a419a120 con 0x7fc2a41084c0 2024-05-26T12:00:44.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc2aa20e700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00d980 tx=0x7fc29c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c017070 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc29c005020 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2a419a410 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c00fa10 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.058+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2a41a66d0 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.059+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2a406a230 con 0x7fc2a41084c0 2024-05-26T12:00:44.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.060+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7fc29c004a30 con 0x7fc2a41084c0 2024-05-26T12:00:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.061+0000 7fc29affd700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 0x7fc2900b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.061+0000 7fc2a9a0d700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 0x7fc2900b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.061+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7fc29c0e2230 con 0x7fc2a41084c0 2024-05-26T12:00:44.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.062+0000 7fc2a9a0d700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 0x7fc2900b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009fd0 tx=0x7fc2a000b7b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:44.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.065+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc29c020020 con 0x7fc2a41084c0 2024-05-26T12:00:44.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.315+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc2a41a69c0 con 0x7fc2a41084c0 2024-05-26T12:00:44.631 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.630+0000 7fc29affd700 1 -- 172.21.15.156:0/630006961 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7fc29c0aa120 con 0x7fc2a41084c0 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 msgr2=0x7fc2900b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 0x7fc2900b4dc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009fd0 tx=0x7fc2a000b7b0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 msgr2=0x7fc2a41999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc29c00d980 tx=0x7fc29c00dc90 comp rx=0 tx=0).stop 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 shutdown_connections 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc2900b2900 0x7fc2900b4dc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc2a41084c0 0x7fc2a41999c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc2a4102c40 0x7fc2a419f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.635 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 --2- 172.21.15.156:0/630006961 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc2a4103600 0x7fc2a419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:44.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 >> 172.21.15.156:0/630006961 conn(0x7fc2a4078ab0 msgr2=0x7fc2a4100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:44.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 shutdown_connections 2024-05-26T12:00:44.636 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:44.634+0000 7fc2abc71700 1 -- 172.21.15.156:0/630006961 wait complete. 2024-05-26T12:00:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:44 smithi186 ceph-mon[36412]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-26T12:00:44.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:44 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-26T12:00:45.023 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:44 smithi156 ceph-mon[32025]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-26T12:00:45.023 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:44 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-26T12:00:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:44 smithi192 ceph-mon[36545]: pgmap v297: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2024-05-26T12:00:45.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:44 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-26T12:00:45.475 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-05-26T12:00:45.475 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs set cephfs max_mds 5 2024-05-26T12:00:45.725 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:45.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:45 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-26T12:00:45.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:45 smithi156 ceph-mon[32025]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:45.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:45 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-26T12:00:45.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:45 smithi186 ceph-mon[36412]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:45 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/630006961' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-26T12:00:46.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:45 smithi192 ceph-mon[36545]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:46.895 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:46 smithi186 ceph-mon[36412]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-26T12:00:47.015 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:46 smithi156 ceph-mon[32025]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-26T12:00:47.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:46 smithi192 ceph-mon[36545]: pgmap v298: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s wr, 4 op/s 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 -- 172.21.15.156:0/1406775364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 msgr2=0x7f22dc108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 --2- 172.21.15.156:0/1406775364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc108900 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f22cc009990 tx=0x7f22cc009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 -- 172.21.15.156:0/1406775364 shutdown_connections 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 --2- 172.21.15.156:0/1406775364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 0x7f22dc10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 --2- 172.21.15.156:0/1406775364 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc108900 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.477 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 --2- 172.21.15.156:0/1406775364 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 0x7f22dc1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 -- 172.21.15.156:0/1406775364 >> 172.21.15.156:0/1406775364 conn(0x7f22dc0793f0 msgr2=0x7f22dc079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 -- 172.21.15.156:0/1406775364 shutdown_connections 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.473+0000 7f22e0aef700 1 -- 172.21.15.156:0/1406775364 wait complete. 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 Processor -- start 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 -- start start 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 0x7f22dc19f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 0x7f22dc19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc1125f0 con 0x7f22dc108520 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc112470 con 0x7f22dc102e00 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22e0aef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f22dc112770 con 0x7f22dc102440 2024-05-26T12:00:47.478 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59228/0 (socket says 172.21.15.156:59228) 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22d9d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 0x7f22dc19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 -- 172.21.15.156:0/3516769079 learned_addr learned my addr 172.21.15.156:0/3516769079 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22da59c700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 0x7f22dc19f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 -- 172.21.15.156:0/3516769079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 msgr2=0x7f22dc19f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 0x7f22dc19f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 -- 172.21.15.156:0/3516769079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 msgr2=0x7f22dc19fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 0x7f22dc19fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.474+0000 7f22dad9d700 1 -- 172.21.15.156:0/3516769079 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22dc19a160 con 0x7f22dc108520 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22dad9d700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22d4007ee0 tx=0x7f22d400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d400dd20 con 0x7f22dc108520 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22dc19a450 con 0x7f22dc108520 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22d4004020 con 0x7f22dc108520 2024-05-26T12:00:47.479 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22d4015660 con 0x7f22dc108520 2024-05-26T12:00:47.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.475+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22dc1a6710 con 0x7f22dc108520 2024-05-26T12:00:47.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.476+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22dc06a230 con 0x7f22dc108520 2024-05-26T12:00:47.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.477+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f22d40041d0 con 0x7f22dc108520 2024-05-26T12:00:47.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.478+0000 7f22cb7fe700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 0x7f22c40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:47.480 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.478+0000 7f22da59c700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 0x7f22c40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:47.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.478+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f22d40e1a10 con 0x7f22dc108520 2024-05-26T12:00:47.481 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.478+0000 7f22da59c700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 0x7f22c40b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b5c0 tx=0x7f22cc005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:47.482 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.482+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f22d4010070 con 0x7f22dc108520 2024-05-26T12:00:47.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:47.732+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f22dc1a68e0 con 0x7f22dc108520 2024-05-26T12:00:48.641 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.640+0000 7f22cb7fe700 1 -- 172.21.15.156:0/3516769079 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f22d40a9880 con 0x7f22dc108520 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 msgr2=0x7f22c40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 0x7f22c40b4d40 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f22cc00b5c0 tx=0x7f22cc005960 comp rx=0 tx=0).stop 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 msgr2=0x7f22dc199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f22d4007ee0 tx=0x7f22d400d3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 shutdown_connections 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f22c40b2880 0x7f22c40b4d40 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:48.644 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f22dc108520 0x7f22dc199a00 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f22dc102440 0x7f22dc19f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 --2- 172.21.15.156:0/3516769079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22dc102e00 0x7f22dc19fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 >> 172.21.15.156:0/3516769079 conn(0x7f22dc0793f0 msgr2=0x7f22dc06b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 shutdown_connections 2024-05-26T12:00:48.645 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:48.643+0000 7f22e0aef700 1 -- 172.21.15.156:0/3516769079 wait complete. 2024-05-26T12:00:48.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:48 smithi186 ceph-mon[36412]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:48.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:48 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-26T12:00:49.032 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:48 smithi156 ceph-mon[32025]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:49.032 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:48 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-26T12:00:49.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:48 smithi192 ceph-mon[36545]: pgmap v299: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:49.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:48 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-05-26T12:00:49.432 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs set cephfs allow_standby_replay true 2024-05-26T12:00:49.683 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:49.927 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-26T12:00:49.928 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:49 smithi156 ceph-mon[32025]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-26T12:00:49.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-26T12:00:49.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-26T12:00:49.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-26T12:00:49.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:49.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-26T12:00:49.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-26T12:00:49.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:49 smithi186 ceph-mon[36412]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3516769079' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-05-26T12:00:50.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:49 smithi192 ceph-mon[36545]: pgmap v300: 97 pgs: 97 active+clean; 579 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2024-05-26T12:00:50.931 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:50 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:active 2024-05-26T12:00:50.931 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:50 smithi156 ceph-mon[32025]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-26T12:00:50.931 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:50 smithi156 ceph-mon[32025]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-26T12:00:50.932 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:50 smithi156 ceph-mon[32025]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-26T12:00:50.932 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:50 smithi156 ceph-mon[32025]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-26T12:00:50.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:50 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:active 2024-05-26T12:00:50.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:50 smithi192 ceph-mon[36545]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-26T12:00:50.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:50 smithi192 ceph-mon[36545]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-26T12:00:50.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:50 smithi192 ceph-mon[36545]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-26T12:00:50.973 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:50 smithi192 ceph-mon[36545]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-26T12:00:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:50 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] up:active 2024-05-26T12:00:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:50 smithi186 ceph-mon[36412]: daemon mds.e assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-26T12:00:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:50 smithi186 ceph-mon[36412]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-05-26T12:00:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:50 smithi186 ceph-mon[36412]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:creating} 9 up:standby 2024-05-26T12:00:50.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:50 smithi186 ceph-mon[36412]: daemon mds.e is now active in filesystem cephfs as rank 2 2024-05-26T12:00:51.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.435+0000 7fb005f70700 1 -- 172.21.15.156:0/3520819168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000103010 msgr2=0x7fb000103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:51.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.435+0000 7fb005f70700 1 --2- 172.21.15.156:0/3520819168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000103010 0x7fb000103490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fafe80099c0 tx=0x7fafe8009cd0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 -- 172.21.15.156:0/3520819168 shutdown_connections 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 --2- 172.21.15.156:0/3520819168 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 0x7fb00010f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 --2- 172.21.15.156:0/3520819168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000108730 0x7fb000108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 --2- 172.21.15.156:0/3520819168 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000103010 0x7fb000103490 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 -- 172.21.15.156:0/3520819168 >> 172.21.15.156:0/3520819168 conn(0x7fb000077780 msgr2=0x7fb000077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 -- 172.21.15.156:0/3520819168 shutdown_connections 2024-05-26T12:00:51.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 -- 172.21.15.156:0/3520819168 wait complete. 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 Processor -- start 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.436+0000 7fb005f70700 1 -- start start 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000103010 0x7fb00019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 0x7fb00019b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb000112760 con 0x7fb000108730 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001125e0 con 0x7fb000103010 2024-05-26T12:00:51.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb0001128e0 con 0x7fb0001039d0 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:59242/0 (socket says 172.21.15.156:59242) 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 -- 172.21.15.156:0/3254361740 learned_addr learned my addr 172.21.15.156:0/3254361740 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fafff7fe700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000103010 0x7fb00019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faff7fff700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 0x7fb00019b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 -- 172.21.15.156:0/3254361740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 msgr2=0x7fb00019b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 0x7fb00019b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 -- 172.21.15.156:0/3254361740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000103010 msgr2=0x7fb00019ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000103010 0x7fb00019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 -- 172.21.15.156:0/3254361740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0001a6200 con 0x7fb000108730 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffffff700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faff80094f0 tx=0x7faff800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:51.440 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8004db0 con 0x7fb000108730 2024-05-26T12:00:51.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faff8009e20 con 0x7fb000108730 2024-05-26T12:00:51.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.437+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0001a63d0 con 0x7fb000108730 2024-05-26T12:00:51.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.438+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faff8015400 con 0x7fb000108730 2024-05-26T12:00:51.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.438+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0001a68e0 con 0x7fb000108730 2024-05-26T12:00:51.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.438+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb00006b830 con 0x7fb000108730 2024-05-26T12:00:51.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.440+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7faff8005260 con 0x7fb000108730 2024-05-26T12:00:51.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.440+0000 7faffd7fa700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 0x7fafec0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:51.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.441+0000 7fafff7fe700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 0x7fafec0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:51.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.441+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7faff80e2280 con 0x7fb000108730 2024-05-26T12:00:51.442 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.441+0000 7fafff7fe700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 0x7fafec0b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faff0005950 tx=0x7faff000b410 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:51.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.444+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7faff8010070 con 0x7fb000108730 2024-05-26T12:00:51.703 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:51.702+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fb0001024a0 con 0x7fb000108730 2024-05-26T12:00:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:active 2024-05-26T12:00:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-26T12:00:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-26T12:00:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:creating} 8 up:standby 2024-05-26T12:00:51.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-05-26T12:00:51.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:51 smithi186 ceph-mon[36412]: pgmap v301: 97 pgs: 97 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2024-05-26T12:00:52.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:active 2024-05-26T12:00:52.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-26T12:00:52.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-26T12:00:52.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:creating} 8 up:standby 2024-05-26T12:00:52.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-05-26T12:00:52.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:51 smithi156 ceph-mon[32025]: pgmap v301: 97 pgs: 97 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] up:active 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: daemon mds.i assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=e=up:active} 9 up:standby 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:creating} 8 up:standby 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: daemon mds.i is now active in filesystem cephfs as rank 3 2024-05-26T12:00:52.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:51 smithi192 ceph-mon[36545]: pgmap v301: 97 pgs: 97 active+clean; 581 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 4 op/s 2024-05-26T12:00:52.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.503+0000 7faffd7fa700 1 -- 172.21.15.156:0/3254361740 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v26) ==== 121+0+0 (secure 0 0 0) 0x7faff80aa0f0 con 0x7fb000108730 2024-05-26T12:00:52.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 msgr2=0x7fafec0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 0x7fafec0b4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7faff0005950 tx=0x7faff000b410 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 msgr2=0x7fb0001a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7faff80094f0 tx=0x7faff800d3b0 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 shutdown_connections 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fafec0b2880 0x7fafec0b4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb000108730 0x7fb0001a5cc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb000103010 0x7fb00019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 --2- 172.21.15.156:0/3254361740 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb0001039d0 0x7fb00019b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 >> 172.21.15.156:0/3254361740 conn(0x7fb000077780 msgr2=0x7fb0000ffff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 shutdown_connections 2024-05-26T12:00:52.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:52.507+0000 7fb005f70700 1 -- 172.21.15.156:0/3254361740 wait complete. 2024-05-26T12:00:52.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:active 2024-05-26T12:00:52.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: Cluster is now healthy 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:active} 8 up:standby 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:active 2024-05-26T12:00:52.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:52 smithi156 ceph-mon[32025]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:active 2024-05-26T12:00:52.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: Cluster is now healthy 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:active} 8 up:standby 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:active 2024-05-26T12:00:52.977 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:52 smithi186 ceph-mon[36412]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:53.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] up:active 2024-05-26T12:00:53.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: daemon mds.k assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-05-26T12:00:53.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-26T12:00:53.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: Cluster is now healthy 2024-05-26T12:00:53.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=e=up:active,3=i=up:active} 8 up:standby 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: daemon mds.k is now active in filesystem cephfs as rank 4 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3254361740' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] up:active 2024-05-26T12:00:53.073 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:52 smithi192 ceph-mon[36545]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:53.292 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs set cephfs session_timeout 300 2024-05-26T12:00:53.540 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:53.913 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:53 smithi156 ceph-mon[32025]: pgmap v302: 97 pgs: 97 active+clean; 582 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:53.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:53 smithi186 ceph-mon[36412]: pgmap v302: 97 pgs: 97 active+clean; 582 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:54.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:53 smithi192 ceph-mon[36545]: pgmap v302: 97 pgs: 97 active+clean; 582 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s rd, 1.2 KiB/s wr, 4 op/s 2024-05-26T12:00:54.976 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: 2024-05-26T12:00:54.647+0000 7f2911fe4700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-05-26T12:00:54.976 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:00:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: 2024-05-26T12:00:54.647+0000 7f2911fe4700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-05-26T12:00:55.237 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 -- 172.21.15.156:0/2508531002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad0101b30 msgr2=0x7f1ad0101fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 --2- 172.21.15.156:0/2508531002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad0101b30 0x7f1ad0101fb0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80099c0 tx=0x7f1ab8009cd0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 -- 172.21.15.156:0/2508531002 shutdown_connections 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 --2- 172.21.15.156:0/2508531002 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad01024f0 0x7f1ad01103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 --2- 172.21.15.156:0/2508531002 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0109520 0x7f1ad0109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 --2- 172.21.15.156:0/2508531002 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad0101b30 0x7f1ad0101fb0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 -- 172.21.15.156:0/2508531002 >> 172.21.15.156:0/2508531002 conn(0x7f1ad0078ab0 msgr2=0x7f1ad00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 -- 172.21.15.156:0/2508531002 shutdown_connections 2024-05-26T12:00:55.238 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.237+0000 7f1ad4889700 1 -- 172.21.15.156:0/2508531002 wait complete. 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 Processor -- start 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 -- start start 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 0x7f1ad0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad0109520 0x7f1ad0103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad0113720 con 0x7f1ad01024f0 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad01135a0 con 0x7f1ad0101b30 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ad4889700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1ad01138a0 con 0x7f1ad0109520 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ac5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:55.240 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ace59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 0x7f1ad0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ac5d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48422/0 (socket says 172.21.15.156:48422) 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1ac5d9b700 1 -- 172.21.15.156:0/1483440905 learned_addr learned my addr 172.21.15.156:0/1483440905 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.238+0000 7f1aced9d700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad0109520 0x7f1ad0103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 -- 172.21.15.156:0/1483440905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad0109520 msgr2=0x7f1ad0103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad0109520 0x7f1ad0103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 -- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 msgr2=0x7f1ad0103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 -- 172.21.15.156:0/1483440905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ad01ac560 con 0x7f1ad0101b30 2024-05-26T12:00:55.241 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ac5d9b700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ace59c700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 0x7f1ad0102d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000d950 tx=0x7f1ac000dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0016070 con 0x7f1ad0101b30 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ad01ac7f0 con 0x7f1ad0101b30 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ac000fd40 con 0x7f1ad0101b30 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ac000e3b0 con 0x7f1ad0101b30 2024-05-26T12:00:55.242 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.239+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ad01acd00 con 0x7f1ad0101b30 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.240+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ad0104e50 con 0x7f1ad0101b30 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.242+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f1ac0019070 con 0x7f1ad0101b30 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.242+0000 7f1ac7fff700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 0x7f1abc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.242+0000 7f1ac5d9b700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 0x7f1abc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.242+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f1ac00e1fd0 con 0x7f1ad0101b30 2024-05-26T12:00:55.243 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.243+0000 7f1ac5d9b700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 0x7f1abc0b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0104560 tx=0x7f1ab8005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:55.246 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.246+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1ac00a9e20 con 0x7f1ad0101b30 2024-05-26T12:00:55.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.499+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f1ad010b470 con 0x7f1ad0101b30 2024-05-26T12:00:55.861 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.860+0000 7f1ac7fff700 1 -- 172.21.15.156:0/1483440905 <== mon.2 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v27) ==== 115+0+0 (secure 0 0 0) 0x7f1ac00a9720 con 0x7f1ad0101b30 2024-05-26T12:00:55.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 msgr2=0x7f1abc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 0x7f1abc0b4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1ad0104560 tx=0x7f1ab8005fb0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 msgr2=0x7f1ad0102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 0x7f1ad0102d10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000d950 tx=0x7f1ac000dc60 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 shutdown_connections 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1abc0b2880 0x7f1abc0b4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1ad0109520 0x7f1ad0103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad0101b30 0x7f1ad0102d10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 --2- 172.21.15.156:0/1483440905 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1ad01024f0 0x7f1ad0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 >> 172.21.15.156:0/1483440905 conn(0x7f1ad0078ab0 msgr2=0x7f1ad0100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 shutdown_connections 2024-05-26T12:00:55.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:55.863+0000 7f1ad4889700 1 -- 172.21.15.156:0/1483440905 wait complete. 2024-05-26T12:00:56.185 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:55 smithi156 ceph-mon[32025]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-05-26T12:00:56.185 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:55 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1483440905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.185 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:55 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:55 smithi186 ceph-mon[36412]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-05-26T12:00:56.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:55 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1483440905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:55 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:55 smithi192 ceph-mon[36545]: pgmap v303: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 3.3 KiB/s wr, 15 op/s 2024-05-26T12:00:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:55 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1483440905' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:55 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-26T12:00:56.645 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-05-26T12:00:56.645 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs subvolumegroup create cephfs qa 2024-05-26T12:00:56.894 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:00:57.182 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:56 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-26T12:00:57.183 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:56 smithi156 ceph-mon[32025]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:57.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:56 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-26T12:00:57.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:56 smithi186 ceph-mon[36412]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:57.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:56 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-26T12:00:57.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:56 smithi192 ceph-mon[36545]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-05-26T12:00:58.154 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:57 smithi186 ceph-mon[36412]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:00:58.250 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:57 smithi156 ceph-mon[32025]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:00:58.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:57 smithi192 ceph-mon[36545]: pgmap v304: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:00:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 -- 172.21.15.156:0/2475220778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 msgr2=0x7f7c601028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 --2- 172.21.15.156:0/2475220778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c601028b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009990 tx=0x7f7c48009ca0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 -- 172.21.15.156:0/2475220778 shutdown_connections 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 --2- 172.21.15.156:0/2475220778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c6010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 --2- 172.21.15.156:0/2475220778 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c601088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 --2- 172.21.15.156:0/2475220778 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c601028b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 -- 172.21.15.156:0/2475220778 >> 172.21.15.156:0/2475220778 conn(0x7f7c600793b0 msgr2=0x7f7c600797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 -- 172.21.15.156:0/2475220778 shutdown_connections 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.610+0000 7f7c65bca700 1 -- 172.21.15.156:0/2475220778 wait complete. 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 Processor -- start 2024-05-26T12:00:58.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 -- start start 2024-05-26T12:00:58.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:58.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c601045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c60104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c60112640 con 0x7f7c60102430 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601124c0 con 0x7f7c60102df0 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c65bca700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7c601127c0 con 0x7f7c60108510 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c5f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c5f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48434/0 (socket says 172.21.15.156:48434) 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c5f7fe700 1 -- 172.21.15.156:0/1999090123 learned_addr learned my addr 172.21.15.156:0/1999090123 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.611+0000 7f7c57fff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c601045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5ffff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c60104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 -- 172.21.15.156:0/1999090123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 msgr2=0x7f7c60104af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c60104af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 -- 172.21.15.156:0/1999090123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 msgr2=0x7f7c601045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c601045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 -- 172.21.15.156:0/1999090123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c601a81b0 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5ffff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c60104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c57fff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c601045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5f7fe700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000b530 tx=0x7f7c5000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50015020 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c50011ce0 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c601a84a0 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c50010d20 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.612+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c601a8990 con 0x7f7c60102430 2024-05-26T12:00:58.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.613+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c6006a230 con 0x7f7c60102430 2024-05-26T12:00:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.614+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 16) ==== 149653+0+0 (secure 0 0 0) 0x7f7c50018020 con 0x7f7c60102430 2024-05-26T12:00:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.615+0000 7f7c5d7fa700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 0x7f7c4c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:00:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.615+0000 7f7c57fff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 0x7f7c4c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:00:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.615+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f7c500e1ec0 con 0x7f7c60102430 2024-05-26T12:00:58.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.616+0000 7f7c57fff700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 0x7f7c4c0b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006010 tx=0x7f7c4800b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:00:58.619 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.618+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7c5001f020 con 0x7f7c60102430 2024-05-26T12:00:58.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.859+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f7c60063980 con 0x7f7c4c0b2880 2024-05-26T12:00:58.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.884+0000 7f7c5d7fa700 1 -- 172.21.15.156:0/1999090123 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f7c60063980 con 0x7f7c4c0b2880 2024-05-26T12:00:58.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 msgr2=0x7f7c4c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 0x7f7c4c0b4d40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7c48006010 tx=0x7f7c4800b540 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 msgr2=0x7f7c60104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7c5000b530 tx=0x7f7c5000b8f0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 shutdown_connections 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c4c0b2880 0x7f7c4c0b4d40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c60108510 0x7f7c60104af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c60102430 0x7f7c60104070 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 --2- 172.21.15.156:0/1999090123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c60102df0 0x7f7c601045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 >> 172.21.15.156:0/1999090123 conn(0x7f7c600793b0 msgr2=0x7f7c6006bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 shutdown_connections 2024-05-26T12:00:58.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:00:58.886+0000 7f7c65bca700 1 -- 172.21.15.156:0/1999090123 wait complete. 2024-05-26T12:00:59.210 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:00:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:00:58.863+0000 7f713c328700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-26T12:00:59.210 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:00:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:00:58.863+0000 7f713c328700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-26T12:00:59.211 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:00:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:00:58.863+0000 7f713c328700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-26T12:00:59.211 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:00:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:00:58.863+0000 7f713c328700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-26T12:00:59.211 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:00:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:00:58.863+0000 7f713c328700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-26T12:00:59.642 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-26T12:00:59.896 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:00.160 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:59 smithi156 ceph-mon[32025]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:01:00.160 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:00:59 smithi156 ceph-mon[32025]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:00.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:59 smithi186 ceph-mon[36412]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:01:00.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:00:59 smithi186 ceph-mon[36412]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:00.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:59 smithi192 ceph-mon[36545]: pgmap v305: 97 pgs: 97 active+clean; 584 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 4.0 KiB/s wr, 18 op/s 2024-05-26T12:01:00.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:00:59 smithi192 ceph-mon[36545]: from='client.14685 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:01.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- 172.21.15.156:0/2362971654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c41028c0 msgr2=0x7f86c4110330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:01.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 --2- 172.21.15.156:0/2362971654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c41028c0 0x7f86c4110330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00b0a0 tx=0x7f86bc00b3b0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- 172.21.15.156:0/2362971654 shutdown_connections 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 --2- 172.21.15.156:0/2362971654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c41028c0 0x7f86c4110330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 --2- 172.21.15.156:0/2362971654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c4108620 0x7f86c4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 --2- 172.21.15.156:0/2362971654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 0x7f86c4102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- 172.21.15.156:0/2362971654 >> 172.21.15.156:0/2362971654 conn(0x7f86c4077780 msgr2=0x7f86c4077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- 172.21.15.156:0/2362971654 shutdown_connections 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- 172.21.15.156:0/2362971654 wait complete. 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 Processor -- start 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.676+0000 7f86cb39d700 1 -- start start 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 0x7f86c41030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c41028c0 0x7f86c4103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86c4113770 con 0x7f86c4108620 2024-05-26T12:01:01.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86c41135f0 con 0x7f86c41028c0 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f86c41138f0 con 0x7f86c4101f00 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48476/0 (socket says 172.21.15.156:48476) 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 -- 172.21.15.156:0/3536351594 learned_addr learned my addr 172.21.15.156:0/3536351594 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c9139700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 0x7f86c41030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c8938700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c41028c0 0x7f86c4103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 -- 172.21.15.156:0/3536351594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 msgr2=0x7f86c41030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 0x7f86c41030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 -- 172.21.15.156:0/3536351594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c41028c0 msgr2=0x7f86c4103620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c41028c0 0x7f86c4103620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 -- 172.21.15.156:0/3536351594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86c41a8150 con 0x7f86c4108620 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86c993a700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00b830 tx=0x7f86bc004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:01.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc021070 con 0x7f86c4108620 2024-05-26T12:01:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86bc007e30 con 0x7f86c4108620 2024-05-26T12:01:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86bc0047d0 con 0x7f86c4108620 2024-05-26T12:01:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86c41a83e0 con 0x7f86c4108620 2024-05-26T12:01:01.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.677+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86c41a8840 con 0x7f86c4108620 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.678+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86c406a230 con 0x7f86c4108620 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.680+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f86bc007980 con 0x7f86c4108620 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.680+0000 7f86ba7fc700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 0x7f86b00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.681+0000 7f86c9139700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 0x7f86b00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.681+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f86bc00f020 con 0x7f86c4108620 2024-05-26T12:01:01.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.681+0000 7f86c9139700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 0x7f86b00b4dd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f86c4078ce0 tx=0x7f86c0007400 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:01.685 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.684+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f86bc02a020 con 0x7f86c4108620 2024-05-26T12:01:01.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.919+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f86c4063980 con 0x7f86b00b2910 2024-05-26T12:01:01.949 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.948+0000 7f86ba7fc700 1 -- 172.21.15.156:0/3536351594 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f86c4063980 con 0x7f86b00b2910 2024-05-26T12:01:01.951 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 msgr2=0x7f86b00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 0x7f86b00b4dd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f86c4078ce0 tx=0x7f86c0007400 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 msgr2=0x7f86c4103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00b830 tx=0x7f86bc004ce0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 shutdown_connections 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86b00b2910 0x7f86b00b4dd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86c4108620 0x7f86c4103b60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86c4101f00 0x7f86c41030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 --2- 172.21.15.156:0/3536351594 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c41028c0 0x7f86c4103620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 >> 172.21.15.156:0/3536351594 conn(0x7f86c4077780 msgr2=0x7f86c4100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 shutdown_connections 2024-05-26T12:01:01.952 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:01.951+0000 7f86cb39d700 1 -- 172.21.15.156:0/3536351594 wait complete. 2024-05-26T12:01:02.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:01 smithi186 ceph-mon[36412]: pgmap v306: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 22 op/s 2024-05-26T12:01:02.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:01 smithi186 ceph-mon[36412]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-26T12:01:02.237 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:01 smithi192 ceph-mon[36545]: pgmap v306: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 22 op/s 2024-05-26T12:01:02.237 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:01 smithi192 ceph-mon[36545]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-26T12:01:02.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:01 smithi156 ceph-mon[32025]: pgmap v306: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 4.7 KiB/s wr, 22 op/s 2024-05-26T12:01:02.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:01 smithi156 ceph-mon[32025]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-26T12:01:02.659 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-26T12:01:02.905 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:03.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:02 smithi186 ceph-mon[36412]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:03.258 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:02 smithi156 ceph-mon[32025]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:03.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:02 smithi192 ceph-mon[36545]: from='client.14697 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:04.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:03 smithi156 ceph-mon[32025]: pgmap v307: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:04.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:03 smithi186 ceph-mon[36412]: pgmap v307: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:04.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:03 smithi192 ceph-mon[36545]: pgmap v307: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 -- 172.21.15.156:0/3055582738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec06c470 msgr2=0x7f63ec1103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 --2- 172.21.15.156:0/3055582738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec06c470 0x7f63ec1103c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f63e400b0a0 tx=0x7f63e400b3b0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 -- 172.21.15.156:0/3055582738 shutdown_connections 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 --2- 172.21.15.156:0/3055582738 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec06c470 0x7f63ec1103c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 --2- 172.21.15.156:0/3055582738 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec1095a0 0x7f63ec109980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 --2- 172.21.15.156:0/3055582738 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06bab0 0x7f63ec06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 -- 172.21.15.156:0/3055582738 >> 172.21.15.156:0/3055582738 conn(0x7f63ec0793b0 msgr2=0x7f63ec0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:04.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 -- 172.21.15.156:0/3055582738 shutdown_connections 2024-05-26T12:01:04.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.582+0000 7f63f2e92700 1 -- 172.21.15.156:0/3055582738 wait complete. 2024-05-26T12:01:04.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 Processor -- start 2024-05-26T12:01:04.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 -- start start 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec06bab0 0x7f63ec19f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06c470 0x7f63ec19fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec113750 con 0x7f63ec1095a0 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec1135d0 con 0x7f63ec06bab0 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f2e92700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f63ec1138d0 con 0x7f63ec06c470 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58360/0 (socket says 172.21.15.156:58360) 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 -- 172.21.15.156:0/419251572 learned_addr learned my addr 172.21.15.156:0/419251572 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63e3fff700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06c470 0x7f63ec19fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f0c2e700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec06bab0 0x7f63ec19f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:04.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 -- 172.21.15.156:0/419251572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06c470 msgr2=0x7f63ec19fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06c470 0x7f63ec19fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 -- 172.21.15.156:0/419251572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec06bab0 msgr2=0x7f63ec19f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec06bab0 0x7f63ec19f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.583+0000 7f63f142f700 1 -- 172.21.15.156:0/419251572 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ec19a230 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63f142f700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f63e400b830 tx=0x7f63e4004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e4021070 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63e4007e30 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e40047d0 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ec19a4c0 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.584+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ec1a67e0 con 0x7f63ec1095a0 2024-05-26T12:01:04.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.585+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63ec10ba80 con 0x7f63ec1095a0 2024-05-26T12:01:04.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.586+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f63e4007980 con 0x7f63ec1095a0 2024-05-26T12:01:04.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.587+0000 7f63e1ffb700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 0x7f63d40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:04.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.588+0000 7f63f0c2e700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 0x7f63d40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:04.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.588+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 7979+0+0 (secure 0 0 0) 0x7f63e400f020 con 0x7f63ec1095a0 2024-05-26T12:01:04.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.588+0000 7f63f0c2e700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 0x7f63d40b4e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:04.592 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.591+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f63e402a020 con 0x7f63ec1095a0 2024-05-26T12:01:04.836 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.835+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f63ec19afc0 con 0x7f63d40b2990 2024-05-26T12:01:04.840 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.840+0000 7f63e1ffb700 1 -- 172.21.15.156:0/419251572 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f63ec19afc0 con 0x7f63d40b2990 2024-05-26T12:01:04.841 INFO:teuthology.orchestra.run.smithi156.stdout:/volumes/qa/sv_0/aaa0625b-5497-4869-8af1-d2f75d9516a0 2024-05-26T12:01:04.843 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 msgr2=0x7f63d40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 0x7f63d40b4e50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f63dc006fd0 tx=0x7f63dc008040 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 msgr2=0x7f63ec199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f63e400b830 tx=0x7f63e4004ce0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 shutdown_connections 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f63d40b2990 0x7f63d40b4e50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f63ec1095a0 0x7f63ec199ad0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63ec06bab0 0x7f63ec19f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 --2- 172.21.15.156:0/419251572 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f63ec06c470 0x7f63ec19fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 >> 172.21.15.156:0/419251572 conn(0x7f63ec0793b0 msgr2=0x7f63ec1011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 shutdown_connections 2024-05-26T12:01:04.844 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:04.843+0000 7f63f2e92700 1 -- 172.21.15.156:0/419251572 wait complete. 2024-05-26T12:01:05.539 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-26T12:01:05.784 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:06.161 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:05 smithi156 ceph-mon[32025]: pgmap v308: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:06.161 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:05 smithi156 ceph-mon[32025]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:06.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:05 smithi186 ceph-mon[36412]: pgmap v308: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:06.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:05 smithi186 ceph-mon[36412]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:06.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:05 smithi192 ceph-mon[36545]: pgmap v308: 97 pgs: 97 active+clean; 591 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.7 KiB/s wr, 20 op/s 2024-05-26T12:01:06.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:05 smithi192 ceph-mon[36545]: from='client.14703 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:07.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-26T12:01:07.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-26T12:01:07.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:06 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-26T12:01:07.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:06 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]: dispatch 2024-05-26T12:01:07.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-05-26T12:01:07.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-05-26T12:01:07.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:06 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]: dispatch 2024-05-26T12:01:07.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 -- 172.21.15.156:0/510595858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 msgr2=0x7f16881104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:07.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 --2- 172.21.15.156:0/510595858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f16881104a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f168000b0d0 tx=0x7f168000b3e0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 -- 172.21.15.156:0/510595858 shutdown_connections 2024-05-26T12:01:07.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 --2- 172.21.15.156:0/510595858 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f16881104a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.510 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 --2- 172.21.15.156:0/510595858 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 0x7f1688103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 --2- 172.21.15.156:0/510595858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 0x7f16881081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 -- 172.21.15.156:0/510595858 >> 172.21.15.156:0/510595858 conn(0x7f168807a620 msgr2=0x7f168807aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:07.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 -- 172.21.15.156:0/510595858 shutdown_connections 2024-05-26T12:01:07.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.509+0000 7f168fd1e700 1 -- 172.21.15.156:0/510595858 wait complete. 2024-05-26T12:01:07.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 Processor -- start 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 -- start start 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 0x7f168819f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 0x7f168819fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1688113710 con 0x7f1688108920 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1688113590 con 0x7f1688104200 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168fd1e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1688113890 con 0x7f1688103850 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168e2bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.510+0000 7f168e2bb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58374/0 (socket says 172.21.15.156:58374) 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 -- 172.21.15.156:0/466143379 learned_addr learned my addr 172.21.15.156:0/466143379 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168daba700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 0x7f168819f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168d2b9700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 0x7f168819fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:07.513 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 -- 172.21.15.156:0/466143379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 msgr2=0x7f168819f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 0x7f168819f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 -- 172.21.15.156:0/466143379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 msgr2=0x7f168819fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 0x7f168819fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 -- 172.21.15.156:0/466143379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f168819a0d0 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168e2bb700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1680009fd0 tx=0x7f1680007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168001f070 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1680007e50 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f168819a360 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f168001b6a0 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.511+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16881a6680 con 0x7f1688108920 2024-05-26T12:01:07.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.512+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f168810baf0 con 0x7f1688108920 2024-05-26T12:01:07.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.513+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f1680022070 con 0x7f1688108920 2024-05-26T12:01:07.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.514+0000 7f167effd700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 0x7f16740b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:07.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.514+0000 7f168daba700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 0x7f16740b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:07.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.514+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8248+0+0 (secure 0 0 0) 0x7f168000f020 con 0x7f1688108920 2024-05-26T12:01:07.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.515+0000 7f168daba700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 0x7f16740b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1684005950 tx=0x7f168400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:07.519 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.519+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1680029020 con 0x7f1688108920 2024-05-26T12:01:07.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:07.756+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1688063980 con 0x7f16740b2910 2024-05-26T12:01:08.229 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: pgmap v309: 97 pgs: 97 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-26T12:01:08.230 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:07 smithi186 ceph-mon[36412]: osdmap e80: 12 total, 12 up, 12 in 2024-05-26T12:01:08.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: pgmap v309: 97 pgs: 97 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-26T12:01:08.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:07 smithi156 ceph-mon[32025]: osdmap e80: 12 total, 12 up, 12 in 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: pgmap v309: 97 pgs: 97 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.5 KiB/s wr, 13 op/s 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.12", "id": [5, 11]}]': finished 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4, 7, 6]}]': finished 2024-05-26T12:01:08.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 6]}]': finished 2024-05-26T12:01:08.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-05-26T12:01:08.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-05-26T12:01:08.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 8]}]': finished 2024-05-26T12:01:08.323 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:07 smithi192 ceph-mon[36545]: osdmap e80: 12 total, 12 up, 12 in 2024-05-26T12:01:08.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.496+0000 7f167effd700 1 -- 172.21.15.156:0/466143379 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1688063980 con 0x7f16740b2910 2024-05-26T12:01:08.499 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 msgr2=0x7f16740b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 0x7f16740b4dd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1684005950 tx=0x7f168400a300 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 msgr2=0x7f1688199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1680009fd0 tx=0x7f1680007a20 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 shutdown_connections 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f16740b2910 0x7f16740b4dd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1688108920 0x7f1688199970 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1688103850 0x7f168819f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 --2- 172.21.15.156:0/466143379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1688104200 0x7f168819fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 >> 172.21.15.156:0/466143379 conn(0x7f168807a620 msgr2=0x7f1688109a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 shutdown_connections 2024-05-26T12:01:08.500 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:08.498+0000 7f168fd1e700 1 -- 172.21.15.156:0/466143379 wait complete. 2024-05-26T12:01:09.142 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:08 smithi156 ceph-mon[32025]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:09.142 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:08 smithi156 ceph-mon[32025]: osdmap e81: 12 total, 12 up, 12 in 2024-05-26T12:01:09.169 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-26T12:01:09.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:08 smithi186 ceph-mon[36412]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:09.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:08 smithi186 ceph-mon[36412]: osdmap e81: 12 total, 12 up, 12 in 2024-05-26T12:01:09.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:08 smithi192 ceph-mon[36545]: from='client.14709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:09.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:08 smithi192 ceph-mon[36545]: osdmap e81: 12 total, 12 up, 12 in 2024-05-26T12:01:09.410 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:10.217 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:09 smithi156 ceph-mon[32025]: pgmap v312: 97 pgs: 1 activating, 2 peering, 94 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 9 op/s; 11 B/s, 0 objects/s recovering 2024-05-26T12:01:10.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:09 smithi186 ceph-mon[36412]: pgmap v312: 97 pgs: 1 activating, 2 peering, 94 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 9 op/s; 11 B/s, 0 objects/s recovering 2024-05-26T12:01:10.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:09 smithi192 ceph-mon[36545]: pgmap v312: 97 pgs: 1 activating, 2 peering, 94 active+clean; 611 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.1 KiB/s wr, 9 op/s; 11 B/s, 0 objects/s recovering 2024-05-26T12:01:11.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.029+0000 7fb27047c700 1 -- 172.21.15.156:0/3770321800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806c670 msgr2=0x7fb268110410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:11.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.029+0000 7fb27047c700 1 --2- 172.21.15.156:0/3770321800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806c670 0x7fb268110410 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fb26000b0a0 tx=0x7fb26000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.067 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- 172.21.15.156:0/3770321800 shutdown_connections 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- 172.21.15.156:0/3770321800 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806c670 0x7fb268110410 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- 172.21.15.156:0/3770321800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb268109620 0x7fb268109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- 172.21.15.156:0/3770321800 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb26806bcb0 0x7fb26806c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- 172.21.15.156:0/3770321800 >> 172.21.15.156:0/3770321800 conn(0x7fb268078490 msgr2=0x7fb2680788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- 172.21.15.156:0/3770321800 shutdown_connections 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- 172.21.15.156:0/3770321800 wait complete. 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 Processor -- start 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- start start 2024-05-26T12:01:11.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb26806c670 0x7fb268102ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb268109620 0x7fb268103020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb2681137d0 con 0x7fb26806bcb0 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb268113650 con 0x7fb26806c670 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.030+0000 7fb27047c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb268113950 con 0x7fb268109620 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58402/0 (socket says 172.21.15.156:58402) 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 -- 172.21.15.156:0/287500390 learned_addr learned my addr 172.21.15.156:0/287500390 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26da17700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb26806c670 0x7fb268102ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26ea19700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb268109620 0x7fb268103020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:11.069 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 -- 172.21.15.156:0/287500390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb268109620 msgr2=0x7fb268103020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb268109620 0x7fb268103020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 -- 172.21.15.156:0/287500390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb26806c670 msgr2=0x7fb268102ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb26806c670 0x7fb268102ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 -- 172.21.15.156:0/287500390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2681a80b0 con 0x7fb26806bcb0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb26e218700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb26400d980 tx=0x7fb26400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb264016070 con 0x7fb26806bcb0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb264004830 con 0x7fb26806bcb0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2640054c0 con 0x7fb26806bcb0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2681a83a0 con 0x7fb26806bcb0 2024-05-26T12:01:11.070 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.031+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2681a8890 con 0x7fb26806bcb0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.032+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb26806a230 con 0x7fb26806bcb0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.034+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fb264004220 con 0x7fb26806bcb0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.035+0000 7fb25f7fe700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 0x7fb2540b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.035+0000 7fb26da17700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 0x7fb2540b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.035+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fb2640e25e0 con 0x7fb26806bcb0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.035+0000 7fb26da17700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 0x7fb2540b4e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb268100ad0 tx=0x7fb25800a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:11.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.038+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb26401f020 con 0x7fb26806bcb0 2024-05-26T12:01:11.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.275+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7fb268063980 con 0x7fb2540b2990 2024-05-26T12:01:11.280 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.280+0000 7fb25f7fe700 1 -- 172.21.15.156:0/287500390 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fb268063980 con 0x7fb2540b2990 2024-05-26T12:01:11.280 INFO:teuthology.orchestra.run.smithi156.stdout:/volumes/qa/sv_1/0a02999c-87e1-4653-8fdd-d61096221daa 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 msgr2=0x7fb2540b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 0x7fb2540b4e50 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb268100ad0 tx=0x7fb25800a380 comp rx=0 tx=0).stop 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 msgr2=0x7fb2681025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb26400d980 tx=0x7fb26400dc90 comp rx=0 tx=0).stop 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 shutdown_connections 2024-05-26T12:01:11.283 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.282+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb2540b2990 0x7fb2540b4e50 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb268109620 0x7fb268103020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb26806bcb0 0x7fb2681025a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 --2- 172.21.15.156:0/287500390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb26806c670 0x7fb268102ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 >> 172.21.15.156:0/287500390 conn(0x7fb268078490 msgr2=0x7fb268101090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 shutdown_connections 2024-05-26T12:01:11.284 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:11.283+0000 7fb27047c700 1 -- 172.21.15.156:0/287500390 wait complete. 2024-05-26T12:01:11.987 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:01:12.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:11 smithi186 ceph-mon[36412]: pgmap v313: 97 pgs: 1 activating, 2 peering, 94 active+clean; 633 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.2 KiB/s wr, 13 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:12.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:11 smithi186 ceph-mon[36412]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:12.227 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:11 smithi156 ceph-mon[32025]: pgmap v313: 97 pgs: 1 activating, 2 peering, 94 active+clean; 633 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.2 KiB/s wr, 13 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:12.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:11 smithi156 ceph-mon[32025]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:11 smithi192 ceph-mon[36545]: pgmap v313: 97 pgs: 1 activating, 2 peering, 94 active+clean; 633 KiB data, 347 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.2 KiB/s wr, 13 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:12.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:11 smithi192 ceph-mon[36545]: from='client.14715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- 172.21.15.156:0/1721472815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 msgr2=0x7f29681028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 --2- 172.21.15.156:0/1721472815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f29681028f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2964009990 tx=0x7f2964009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- 172.21.15.156:0/1721472815 shutdown_connections 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 --2- 172.21.15.156:0/1721472815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 --2- 172.21.15.156:0/1721472815 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 0x7f2968108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 --2- 172.21.15.156:0/1721472815 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f29681028f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- 172.21.15.156:0/1721472815 >> 172.21.15.156:0/1721472815 conn(0x7f2968079410 msgr2=0x7f2968079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- 172.21.15.156:0/1721472815 shutdown_connections 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- 172.21.15.156:0/1721472815 wait complete. 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 Processor -- start 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.860+0000 7f29705a5700 1 -- start start 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 0x7f2968199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2968112640 con 0x7f2968102470 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f29681124c0 con 0x7f2968102e30 2024-05-26T12:01:13.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f29681127c0 con 0x7f29681084b0 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:58414/0 (socket says 172.21.15.156:58414) 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 -- 172.21.15.156:0/3161123016 learned_addr learned my addr 172.21.15.156:0/3161123016 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296db40700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296eb42700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 0x7f2968199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 -- 172.21.15.156:0/3161123016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 msgr2=0x7f2968199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 0x7f2968199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 -- 172.21.15.156:0/3161123016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 msgr2=0x7f296819fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296819fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 -- 172.21.15.156:0/3161123016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296819a210 con 0x7f2968102470 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296db40700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296819fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f296e341700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f295800b530 tx=0x7f295800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:13.876 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958015020 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296819a500 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2958011ce0 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958010d20 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.861+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29681a67c0 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.862+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f296806a230 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.864+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f2958018020 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.864+0000 7f295f7fe700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 0x7f29540b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.865+0000 7f296db40700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 0x7f29540b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.865+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f29580e2000 con 0x7f2968102470 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.865+0000 7f296db40700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 0x7f29540b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2964006010 tx=0x7f296400b540 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:13.877 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:13.868+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f295801f020 con 0x7f2968102470 2024-05-26T12:01:14.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.124+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f29681a6ab0 con 0x7f2968102470 2024-05-26T12:01:14.125 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.125+0000 7f295f7fe700 1 -- 172.21.15.156:0/3161123016 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7f29580a9d60 con 0x7f2968102470 2024-05-26T12:01:14.126 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:14.126 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 msgr2=0x7f29540b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 0x7f29540b4dd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2964006010 tx=0x7f296400b540 comp rx=0 tx=0).stop 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 msgr2=0x7f296819f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f295800b530 tx=0x7f295800b8f0 comp rx=0 tx=0).stop 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 shutdown_connections 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f29540b2910 0x7f29540b4dd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f29681084b0 0x7f2968199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2968102470 0x7f296819f8f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 --2- 172.21.15.156:0/3161123016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2968102e30 0x7f296819fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 >> 172.21.15.156:0/3161123016 conn(0x7f2968079410 msgr2=0x7f2968106320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 shutdown_connections 2024-05-26T12:01:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:14.127+0000 7f29705a5700 1 -- 172.21.15.156:0/3161123016 wait complete. 2024-05-26T12:01:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:01:14.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:13 smithi186 ceph-mon[36412]: pgmap v314: 97 pgs: 1 activating, 1 peering, 95 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 11 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:14.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:13 smithi156 ceph-mon[32025]: pgmap v314: 97 pgs: 1 activating, 1 peering, 95 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 11 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:14.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:13 smithi192 ceph-mon[36545]: pgmap v314: 97 pgs: 1 activating, 1 peering, 95 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.1 KiB/s wr, 11 op/s; 3.4 KiB/s, 0 keys/s, 1 objects/s recovering 2024-05-26T12:01:14.792 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:01:15.036 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:15.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:14 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3161123016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:01:15.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:14 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3161123016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:01:15.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:14 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3161123016' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:01:16.187 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:15 smithi156 ceph-mon[32025]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.5 KiB/s wr, 10 op/s; 3.4 KiB/s, 2 keys/s, 2 objects/s recovering 2024-05-26T12:01:16.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:15 smithi186 ceph-mon[36412]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.5 KiB/s wr, 10 op/s; 3.4 KiB/s, 2 keys/s, 2 objects/s recovering 2024-05-26T12:01:16.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:15 smithi192 ceph-mon[36545]: pgmap v315: 97 pgs: 97 active+clean; 633 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.5 KiB/s wr, 10 op/s; 3.4 KiB/s, 2 keys/s, 2 objects/s recovering 2024-05-26T12:01:16.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 -- 172.21.15.156:0/4171419849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940108520 msgr2=0x7f9940108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 --2- 172.21.15.156:0/4171419849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940108520 0x7f9940108900 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f993c009990 tx=0x7f993c009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 -- 172.21.15.156:0/4171419849 shutdown_connections 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 --2- 172.21.15.156:0/4171419849 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102e00 0x7f994010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 --2- 172.21.15.156:0/4171419849 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940108520 0x7f9940108900 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 --2- 172.21.15.156:0/4171419849 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940102440 0x7f99401028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 -- 172.21.15.156:0/4171419849 >> 172.21.15.156:0/4171419849 conn(0x7f99400793f0 msgr2=0x7f9940079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 -- 172.21.15.156:0/4171419849 shutdown_connections 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.754+0000 7f99485f2700 1 -- 172.21.15.156:0/4171419849 wait complete. 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 Processor -- start 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 -- start start 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102440 0x7f9940199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:16.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940108520 0x7f994019e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f99401125d0 con 0x7f9940102e00 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f9940112450 con 0x7f9940108520 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f99485f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f9940112750 con 0x7f9940102440 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f9945b8d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f9945b8d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41098/0 (socket says 172.21.15.156:41098) 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f9945b8d700 1 -- 172.21.15.156:0/3602438615 learned_addr learned my addr 172.21.15.156:0/3602438615 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f994638e700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102440 0x7f9940199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.755+0000 7f9946b8f700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940108520 0x7f994019e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 -- 172.21.15.156:0/3602438615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102440 msgr2=0x7f9940199c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102440 0x7f9940199c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.758 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 -- 172.21.15.156:0/3602438615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940108520 msgr2=0x7f994019e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940108520 0x7f994019e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 -- 172.21.15.156:0/3602438615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f994019ec50 con 0x7f9940102e00 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f9945b8d700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f993000c920 tx=0x7f993000cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930004a30 con 0x7f9940102e00 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99300054d0 con 0x7f9940102e00 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f994019ef40 con 0x7f9940102e00 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930007c70 con 0x7f9940102e00 2024-05-26T12:01:16.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.756+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9940103fb0 con 0x7f9940102e00 2024-05-26T12:01:16.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.757+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f994006a230 con 0x7f9940102e00 2024-05-26T12:01:16.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.758+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f9930004c60 con 0x7f9940102e00 2024-05-26T12:01:16.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.759+0000 7f99377fe700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 0x7f992c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:16.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.760+0000 7f994638e700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 0x7f992c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:16.760 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.760+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f99300e22d0 con 0x7f9940102e00 2024-05-26T12:01:16.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.760+0000 7f994638e700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 0x7f992c0b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f993c00b5c0 tx=0x7f993c005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:16.763 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.763+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9930014030 con 0x7f9940102e00 2024-05-26T12:01:16.972 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.971+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f994019afb0 con 0x7f9940102e00 2024-05-26T12:01:16.973 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.973+0000 7f99377fe700 1 -- 172.21.15.156:0/3602438615 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22707 (secure 0 0 0) 0x7f99300a9e80 con 0x7f9940102e00 2024-05-26T12:01:16.974 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:16.974 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":81,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:01:07.937226+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 msgr2=0x7f992c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 0x7f992c0b4dd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f993c00b5c0 tx=0x7f993c005960 comp rx=0 tx=0).stop 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 msgr2=0x7f994019a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f993000c920 tx=0x7f993000cc30 comp rx=0 tx=0).stop 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 shutdown_connections 2024-05-26T12:01:16.976 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f992c0b2910 0x7f992c0b4dd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9940108520 0x7f994019e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f9940102440 0x7f9940199c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 --2- 172.21.15.156:0/3602438615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f9940102e00 0x7f994019a190 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.975+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 >> 172.21.15.156:0/3602438615 conn(0x7f99400793f0 msgr2=0x7f994006b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.976+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 shutdown_connections 2024-05-26T12:01:16.977 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:16.976+0000 7f99485f2700 1 -- 172.21.15.156:0/3602438615 wait complete. 2024-05-26T12:01:17.675 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-26T12:01:17.675 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-26T12:01:17.915 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:18.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:17 smithi186 ceph-mon[36412]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 11 op/s; 2.8 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:18.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:17 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3602438615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:18.276 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:17 smithi156 ceph-mon[32025]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 11 op/s; 2.8 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:18.276 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:17 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3602438615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:18.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:17 smithi192 ceph-mon[36545]: pgmap v316: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 3.5 KiB/s wr, 11 op/s; 2.8 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:18.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:17 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3602438615' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 -- 172.21.15.156:0/3381512357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac1095e0 msgr2=0x7f71ac1099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 --2- 172.21.15.156:0/3381512357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac1095e0 0x7f71ac1099c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f71a00099c0 tx=0x7f71a0009cd0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 -- 172.21.15.156:0/3381512357 shutdown_connections 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 --2- 172.21.15.156:0/3381512357 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac103fc0 0x7f71ac1103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 --2- 172.21.15.156:0/3381512357 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac1095e0 0x7f71ac1099c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.600 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 --2- 172.21.15.156:0/3381512357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103600 0x7f71ac103a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 -- 172.21.15.156:0/3381512357 >> 172.21.15.156:0/3381512357 conn(0x7f71ac07a700 msgr2=0x7f71ac07ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 -- 172.21.15.156:0/3381512357 shutdown_connections 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.597+0000 7f71b2081700 1 -- 172.21.15.156:0/3381512357 wait complete. 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 Processor -- start 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 -- start start 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103fc0 0x7f71ac19fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac1095e0 0x7f71ac199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac113740 con 0x7f71ac103600 2024-05-26T12:01:19.601 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1135c0 con 0x7f71ac103fc0 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71b2081700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f71ac1138c0 con 0x7f71ac1095e0 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71ab7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71ab7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41108/0 (socket says 172.21.15.156:41108) 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.598+0000 7f71ab7fe700 1 -- 172.21.15.156:0/1992405417 learned_addr learned my addr 172.21.15.156:0/1992405417 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71aaffd700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103fc0 0x7f71ac19fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71abfff700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac1095e0 0x7f71ac199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 -- 172.21.15.156:0/1992405417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac1095e0 msgr2=0x7f71ac199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac1095e0 0x7f71ac199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 -- 172.21.15.156:0/1992405417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103fc0 msgr2=0x7f71ac19fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103fc0 0x7f71ac19fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.602 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 -- 172.21.15.156:0/1992405417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac19a1c0 con 0x7f71ac103600 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71ab7fe700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f71a000b580 tx=0x7f71a00037c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a0027070 con 0x7f71ac103600 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac19a450 con 0x7f71ac103600 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f71a0004110 con 0x7f71ac103600 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71a000f460 con 0x7f71ac103600 2024-05-26T12:01:19.603 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.599+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71ac1a6770 con 0x7f71ac103600 2024-05-26T12:01:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.600+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71ac10ba70 con 0x7f71ac103600 2024-05-26T12:01:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.601+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f71a002a070 con 0x7f71ac103600 2024-05-26T12:01:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.602+0000 7f71a8ff9700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 0x7f71940b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.602+0000 7f71aaffd700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 0x7f71940b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:19.604 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.602+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f71a00e37a0 con 0x7f71ac103600 2024-05-26T12:01:19.605 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.603+0000 7f71aaffd700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 0x7f71940b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f719c005fd0 tx=0x7f719c00a560 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:19.606 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.606+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f71a0031070 con 0x7f71ac103600 2024-05-26T12:01:19.808 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.807+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f71ac19af90 con 0x7f71940b2910 2024-05-26T12:01:19.816 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.815+0000 7f71a8ff9700 1 -- 172.21.15.156:0/1992405417 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258997 (secure 0 0 0) 0x7f71ac19af90 con 0x7f71940b2910 2024-05-26T12:01:19.820 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:19.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 msgr2=0x7f71940b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:19.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 0x7f71940b4dd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f719c005fd0 tx=0x7f719c00a560 comp rx=0 tx=0).stop 2024-05-26T12:01:19.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 msgr2=0x7f71ac19f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:19.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f71a000b580 tx=0x7f71a00037c0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 shutdown_connections 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f71940b2910 0x7f71940b4dd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f71ac1095e0 0x7f71ac199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f71ac103600 0x7f71ac19f8a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 --2- 172.21.15.156:0/1992405417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71ac103fc0 0x7f71ac19fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 >> 172.21.15.156:0/1992405417 conn(0x7f71ac07a700 msgr2=0x7f71ac1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 shutdown_connections 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:19.818+0000 7f71b2081700 1 -- 172.21.15.156:0/1992405417 wait complete. 2024-05-26T12:01:19.823 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-05-26T12:01:20.131 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:19 smithi156 ceph-mon[32025]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 10 op/s; 2.5 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:20.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:19 smithi186 ceph-mon[36412]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 10 op/s; 2.5 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:20.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:19 smithi192 ceph-mon[36545]: pgmap v317: 97 pgs: 97 active+clean; 638 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 3.2 KiB/s wr, 10 op/s; 2.5 KiB/s, 1 keys/s, 2 objects/s recovering 2024-05-26T12:01:20.535 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":317,"stamp":"2024-05-26T12:01:18.805585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":652920,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":338,"num_read_kb":258,"num_write":364,"num_write_kb":2281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2980013,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":300,"ondisk_log_size":300,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355884,"kb_used_data":10864,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585940,"statfs":{"total":1151940427776,"available":1151576002560,"internally_reserved":0,"allocated":11124736,"data_stored":7086566,"data_compressed":182304,"data_compressed_allocated":241664,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":27539,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":66,"num_write":20,"num_write_kb":35,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":28173,"num_keys_recovered":17,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.866729"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942670+0000","last_change":"2024-05-26T12:00:32.620823+0000","last_active":"2024-05-26T12:01:07.942670+0000","last_peered":"2024-05-26T12:01:07.942670+0000","last_clean":"2024-05-26T12:01:07.942670+0000","last_became_active":"2024-05-26T12:00:32.619171+0000","last_became_peered":"2024-05-26T12:00:32.619171+0000","last_unstale":"2024-05-26T12:01:07.942670+0000","last_undegraded":"2024-05-26T12:01:07.942670+0000","last_fullsized":"2024-05-26T12:01:07.942670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:57:59.459768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940886+0000","last_change":"2024-05-26T12:00:32.618888+0000","last_active":"2024-05-26T12:01:07.940886+0000","last_peered":"2024-05-26T12:01:07.940886+0000","last_clean":"2024-05-26T12:01:07.940886+0000","last_became_active":"2024-05-26T12:00:32.618235+0000","last_became_peered":"2024-05-26T12:00:32.618235+0000","last_unstale":"2024-05-26T12:01:07.940886+0000","last_undegraded":"2024-05-26T12:01:07.940886+0000","last_fullsized":"2024-05-26T12:01:07.940886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:56:08.852594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940789+0000","last_change":"2024-05-26T12:00:32.613694+0000","last_active":"2024-05-26T12:01:07.940789+0000","last_peered":"2024-05-26T12:01:07.940789+0000","last_clean":"2024-05-26T12:01:07.940789+0000","last_became_active":"2024-05-26T12:00:32.613389+0000","last_became_peered":"2024-05-26T12:00:32.613389+0000","last_unstale":"2024-05-26T12:01:07.940789+0000","last_undegraded":"2024-05-26T12:01:07.940789+0000","last_fullsized":"2024-05-26T12:01:07.940789+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:04.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.471139+0000","last_change":"2024-05-26T12:00:32.620149+0000","last_active":"2024-05-26T12:01:08.471139+0000","last_peered":"2024-05-26T12:01:08.471139+0000","last_clean":"2024-05-26T12:01:08.471139+0000","last_became_active":"2024-05-26T12:00:32.614534+0000","last_became_peered":"2024-05-26T12:00:32.614534+0000","last_unstale":"2024-05-26T12:01:08.471139+0000","last_undegraded":"2024-05-26T12:01:08.471139+0000","last_fullsized":"2024-05-26T12:01:08.471139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:24:41.144826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945382+0000","last_change":"2024-05-26T12:00:32.605841+0000","last_active":"2024-05-26T12:01:07.945382+0000","last_peered":"2024-05-26T12:01:07.945382+0000","last_clean":"2024-05-26T12:01:07.945382+0000","last_became_active":"2024-05-26T12:00:32.605438+0000","last_became_peered":"2024-05-26T12:00:32.605438+0000","last_unstale":"2024-05-26T12:01:07.945382+0000","last_undegraded":"2024-05-26T12:01:07.945382+0000","last_fullsized":"2024-05-26T12:01:07.945382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:05:01.341398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944312+0000","last_change":"2024-05-26T12:00:32.625270+0000","last_active":"2024-05-26T12:01:07.944312+0000","last_peered":"2024-05-26T12:01:07.944312+0000","last_clean":"2024-05-26T12:01:07.944312+0000","last_became_active":"2024-05-26T12:00:32.624892+0000","last_became_peered":"2024-05-26T12:00:32.624892+0000","last_unstale":"2024-05-26T12:01:07.944312+0000","last_undegraded":"2024-05-26T12:01:07.944312+0000","last_fullsized":"2024-05-26T12:01:07.944312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:44.581793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.943961+0000","last_change":"2024-05-26T12:00:32.613286+0000","last_active":"2024-05-26T12:01:07.943961+0000","last_peered":"2024-05-26T12:01:07.943961+0000","last_clean":"2024-05-26T12:01:07.943961+0000","last_became_active":"2024-05-26T12:00:32.612896+0000","last_became_peered":"2024-05-26T12:00:32.612896+0000","last_unstale":"2024-05-26T12:01:07.943961+0000","last_undegraded":"2024-05-26T12:01:07.943961+0000","last_fullsized":"2024-05-26T12:01:07.943961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:34:29.796529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939870+0000","last_change":"2024-05-26T12:00:32.620975+0000","last_active":"2024-05-26T12:01:07.939870+0000","last_peered":"2024-05-26T12:01:07.939870+0000","last_clean":"2024-05-26T12:01:07.939870+0000","last_became_active":"2024-05-26T12:00:32.618576+0000","last_became_peered":"2024-05-26T12:00:32.618576+0000","last_unstale":"2024-05-26T12:01:07.939870+0000","last_undegraded":"2024-05-26T12:01:07.939870+0000","last_fullsized":"2024-05-26T12:01:07.939870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:51:04.187910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947841+0000","last_change":"2024-05-26T12:00:32.618617+0000","last_active":"2024-05-26T12:01:07.947841+0000","last_peered":"2024-05-26T12:01:07.947841+0000","last_clean":"2024-05-26T12:01:07.947841+0000","last_became_active":"2024-05-26T12:00:32.617354+0000","last_became_peered":"2024-05-26T12:00:32.617354+0000","last_unstale":"2024-05-26T12:01:07.947841+0000","last_undegraded":"2024-05-26T12:01:07.947841+0000","last_fullsized":"2024-05-26T12:01:07.947841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:04:22.469952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944741+0000","last_change":"2024-05-26T12:00:32.618169+0000","last_active":"2024-05-26T12:01:07.944741+0000","last_peered":"2024-05-26T12:01:07.944741+0000","last_clean":"2024-05-26T12:01:07.944741+0000","last_became_active":"2024-05-26T12:00:32.617709+0000","last_became_peered":"2024-05-26T12:00:32.617709+0000","last_unstale":"2024-05-26T12:01:07.944741+0000","last_undegraded":"2024-05-26T12:01:07.944741+0000","last_fullsized":"2024-05-26T12:01:07.944741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:21.801512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945015+0000","last_change":"2024-05-26T12:00:32.623233+0000","last_active":"2024-05-26T12:01:07.945015+0000","last_peered":"2024-05-26T12:01:07.945015+0000","last_clean":"2024-05-26T12:01:07.945015+0000","last_became_active":"2024-05-26T12:00:32.622483+0000","last_became_peered":"2024-05-26T12:00:32.622483+0000","last_unstale":"2024-05-26T12:01:07.945015+0000","last_undegraded":"2024-05-26T12:01:07.945015+0000","last_fullsized":"2024-05-26T12:01:07.945015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:14.784275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.471103+0000","last_change":"2024-05-26T12:00:32.614575+0000","last_active":"2024-05-26T12:01:08.471103+0000","last_peered":"2024-05-26T12:01:08.471103+0000","last_clean":"2024-05-26T12:01:08.471103+0000","last_became_active":"2024-05-26T12:00:32.614020+0000","last_became_peered":"2024-05-26T12:00:32.614020+0000","last_unstale":"2024-05-26T12:01:08.471103+0000","last_undegraded":"2024-05-26T12:01:08.471103+0000","last_fullsized":"2024-05-26T12:01:08.471103+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:33:05.747419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946775+0000","last_change":"2024-05-26T12:00:32.616624+0000","last_active":"2024-05-26T12:01:07.946775+0000","last_peered":"2024-05-26T12:01:07.946775+0000","last_clean":"2024-05-26T12:01:07.946775+0000","last_became_active":"2024-05-26T12:00:32.616280+0000","last_became_peered":"2024-05-26T12:00:32.616280+0000","last_unstale":"2024-05-26T12:01:07.946775+0000","last_undegraded":"2024-05-26T12:01:07.946775+0000","last_fullsized":"2024-05-26T12:01:07.946775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:45:02.466249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940839+0000","last_change":"2024-05-26T12:00:32.617058+0000","last_active":"2024-05-26T12:01:07.940839+0000","last_peered":"2024-05-26T12:01:07.940839+0000","last_clean":"2024-05-26T12:01:07.940839+0000","last_became_active":"2024-05-26T12:00:32.614651+0000","last_became_peered":"2024-05-26T12:00:32.614651+0000","last_unstale":"2024-05-26T12:01:07.940839+0000","last_undegraded":"2024-05-26T12:01:07.940839+0000","last_fullsized":"2024-05-26T12:01:07.940839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:38:50.234745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.950685+0000","last_change":"2024-05-26T12:01:07.950685+0000","last_active":"2024-05-26T12:01:07.950685+0000","last_peered":"2024-05-26T12:01:07.950685+0000","last_clean":"2024-05-26T12:01:07.950685+0000","last_became_active":"2024-05-26T12:01:07.950383+0000","last_became_peered":"2024-05-26T12:01:07.950383+0000","last_unstale":"2024-05-26T12:01:07.950685+0000","last_undegraded":"2024-05-26T12:01:07.950685+0000","last_fullsized":"2024-05-26T12:01:07.950685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:02:42.832274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947055+0000","last_change":"2024-05-26T12:00:32.618850+0000","last_active":"2024-05-26T12:01:07.947055+0000","last_peered":"2024-05-26T12:01:07.947055+0000","last_clean":"2024-05-26T12:01:07.947055+0000","last_became_active":"2024-05-26T12:00:32.618494+0000","last_became_peered":"2024-05-26T12:00:32.618494+0000","last_unstale":"2024-05-26T12:01:07.947055+0000","last_undegraded":"2024-05-26T12:01:07.947055+0000","last_fullsized":"2024-05-26T12:01:07.947055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:23:43.555380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942836+0000","last_change":"2024-05-26T12:00:32.621515+0000","last_active":"2024-05-26T12:01:07.942836+0000","last_peered":"2024-05-26T12:01:07.942836+0000","last_clean":"2024-05-26T12:01:07.942836+0000","last_became_active":"2024-05-26T12:00:32.621091+0000","last_became_peered":"2024-05-26T12:00:32.621091+0000","last_unstale":"2024-05-26T12:01:07.942836+0000","last_undegraded":"2024-05-26T12:01:07.942836+0000","last_fullsized":"2024-05-26T12:01:07.942836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:32:07.614386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942309+0000","last_change":"2024-05-26T12:00:32.612371+0000","last_active":"2024-05-26T12:01:07.942309+0000","last_peered":"2024-05-26T12:01:07.942309+0000","last_clean":"2024-05-26T12:01:07.942309+0000","last_became_active":"2024-05-26T12:00:32.612082+0000","last_became_peered":"2024-05-26T12:00:32.612082+0000","last_unstale":"2024-05-26T12:01:07.942309+0000","last_undegraded":"2024-05-26T12:01:07.942309+0000","last_fullsized":"2024-05-26T12:01:07.942309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:25:15.850251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.949132+0000","last_change":"2024-05-26T12:00:32.611421+0000","last_active":"2024-05-26T12:01:07.949132+0000","last_peered":"2024-05-26T12:01:07.949132+0000","last_clean":"2024-05-26T12:01:07.949132+0000","last_became_active":"2024-05-26T12:00:32.609716+0000","last_became_peered":"2024-05-26T12:00:32.609716+0000","last_unstale":"2024-05-26T12:01:07.949132+0000","last_undegraded":"2024-05-26T12:01:07.949132+0000","last_fullsized":"2024-05-26T12:01:07.949132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:55.197945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":62,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:12.539221+0000","last_change":"2024-05-26T12:00:28.578034+0000","last_active":"2024-05-26T12:01:12.539221+0000","last_peered":"2024-05-26T12:01:12.539221+0000","last_clean":"2024-05-26T12:01:12.539221+0000","last_became_active":"2024-05-26T12:00:28.577682+0000","last_became_peered":"2024-05-26T12:00:28.577682+0000","last_unstale":"2024-05-26T12:01:12.539221+0000","last_undegraded":"2024-05-26T12:01:12.539221+0000","last_fullsized":"2024-05-26T12:01:12.539221+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:27:11.135886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":23,"num_read_kb":23,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940258+0000","last_change":"2024-05-26T12:00:32.609857+0000","last_active":"2024-05-26T12:01:07.940258+0000","last_peered":"2024-05-26T12:01:07.940258+0000","last_clean":"2024-05-26T12:01:07.940258+0000","last_became_active":"2024-05-26T12:00:32.609380+0000","last_became_peered":"2024-05-26T12:00:32.609380+0000","last_unstale":"2024-05-26T12:01:07.940258+0000","last_undegraded":"2024-05-26T12:01:07.940258+0000","last_fullsized":"2024-05-26T12:01:07.940258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:39:04.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940461+0000","last_change":"2024-05-26T12:00:28.671940+0000","last_active":"2024-05-26T12:01:07.940461+0000","last_peered":"2024-05-26T12:01:07.940461+0000","last_clean":"2024-05-26T12:01:07.940461+0000","last_became_active":"2024-05-26T12:00:28.671602+0000","last_became_peered":"2024-05-26T12:00:28.671602+0000","last_unstale":"2024-05-26T12:01:07.940461+0000","last_undegraded":"2024-05-26T12:01:07.940461+0000","last_fullsized":"2024-05-26T12:01:07.940461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:38:52.069136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941409+0000","last_change":"2024-05-26T12:00:32.613900+0000","last_active":"2024-05-26T12:01:07.941409+0000","last_peered":"2024-05-26T12:01:07.941409+0000","last_clean":"2024-05-26T12:01:07.941409+0000","last_became_active":"2024-05-26T12:00:32.610264+0000","last_became_peered":"2024-05-26T12:00:32.610264+0000","last_unstale":"2024-05-26T12:01:07.941409+0000","last_undegraded":"2024-05-26T12:01:07.941409+0000","last_fullsized":"2024-05-26T12:01:07.941409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:28:58.712711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.483676+0000","last_change":"2024-05-26T12:01:08.483676+0000","last_active":"2024-05-26T12:01:08.483676+0000","last_peered":"2024-05-26T12:01:08.483676+0000","last_clean":"2024-05-26T12:01:08.483676+0000","last_became_active":"2024-05-26T12:01:08.471437+0000","last_became_peered":"2024-05-26T12:01:08.471437+0000","last_unstale":"2024-05-26T12:01:08.483676+0000","last_undegraded":"2024-05-26T12:01:08.483676+0000","last_fullsized":"2024-05-26T12:01:08.483676+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:43:38.663368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947387+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:01:07.947387+0000","last_peered":"2024-05-26T12:01:07.947387+0000","last_clean":"2024-05-26T12:01:07.947387+0000","last_became_active":"2024-05-26T12:00:32.617702+0000","last_became_peered":"2024-05-26T12:00:32.617702+0000","last_unstale":"2024-05-26T12:01:07.947387+0000","last_undegraded":"2024-05-26T12:01:07.947387+0000","last_fullsized":"2024-05-26T12:01:07.947387+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:25:06.527677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946377+0000","last_change":"2024-05-26T12:00:28.575648+0000","last_active":"2024-05-26T12:01:07.946377+0000","last_peered":"2024-05-26T12:01:07.946377+0000","last_clean":"2024-05-26T12:01:07.946377+0000","last_became_active":"2024-05-26T12:00:28.575170+0000","last_became_peered":"2024-05-26T12:00:28.575170+0000","last_unstale":"2024-05-26T12:01:07.946377+0000","last_undegraded":"2024-05-26T12:01:07.946377+0000","last_fullsized":"2024-05-26T12:01:07.946377+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:09:29.213605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945505+0000","last_change":"2024-05-26T12:00:32.617825+0000","last_active":"2024-05-26T12:01:07.945505+0000","last_peered":"2024-05-26T12:01:07.945505+0000","last_clean":"2024-05-26T12:01:07.945505+0000","last_became_active":"2024-05-26T12:00:32.617517+0000","last_became_peered":"2024-05-26T12:00:32.617517+0000","last_unstale":"2024-05-26T12:01:07.945505+0000","last_undegraded":"2024-05-26T12:01:07.945505+0000","last_fullsized":"2024-05-26T12:01:07.945505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:36:33.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:12.540393+0000","last_change":"2024-05-26T12:00:28.584639+0000","last_active":"2024-05-26T12:01:12.540393+0000","last_peered":"2024-05-26T12:01:12.540393+0000","last_clean":"2024-05-26T12:01:12.540393+0000","last_became_active":"2024-05-26T12:00:28.583063+0000","last_became_peered":"2024-05-26T12:00:28.583063+0000","last_unstale":"2024-05-26T12:01:12.540393+0000","last_undegraded":"2024-05-26T12:01:12.540393+0000","last_fullsized":"2024-05-26T12:01:12.540393+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:01:25.804547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946260+0000","last_change":"2024-05-26T12:00:32.609117+0000","last_active":"2024-05-26T12:01:07.946260+0000","last_peered":"2024-05-26T12:01:07.946260+0000","last_clean":"2024-05-26T12:01:07.946260+0000","last_became_active":"2024-05-26T12:00:32.608770+0000","last_became_peered":"2024-05-26T12:00:32.608770+0000","last_unstale":"2024-05-26T12:01:07.946260+0000","last_undegraded":"2024-05-26T12:01:07.946260+0000","last_fullsized":"2024-05-26T12:01:07.946260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:19:57.422101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.954880+0000","last_change":"2024-05-26T12:01:07.954880+0000","last_active":"2024-05-26T12:01:07.954880+0000","last_peered":"2024-05-26T12:01:07.954880+0000","last_clean":"2024-05-26T12:01:07.954880+0000","last_became_active":"2024-05-26T12:01:07.949506+0000","last_became_peered":"2024-05-26T12:01:07.949506+0000","last_unstale":"2024-05-26T12:01:07.954880+0000","last_undegraded":"2024-05-26T12:01:07.954880+0000","last_fullsized":"2024-05-26T12:01:07.954880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:40:37.041619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945386+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:01:07.945386+0000","last_peered":"2024-05-26T12:01:07.945386+0000","last_clean":"2024-05-26T12:01:07.945386+0000","last_became_active":"2024-05-26T12:00:32.615691+0000","last_became_peered":"2024-05-26T12:00:32.615691+0000","last_unstale":"2024-05-26T12:01:07.945386+0000","last_undegraded":"2024-05-26T12:01:07.945386+0000","last_fullsized":"2024-05-26T12:01:07.945386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:58:26.245149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946543+0000","last_change":"2024-05-26T12:00:28.571893+0000","last_active":"2024-05-26T12:01:07.946543+0000","last_peered":"2024-05-26T12:01:07.946543+0000","last_clean":"2024-05-26T12:01:07.946543+0000","last_became_active":"2024-05-26T12:00:28.571502+0000","last_became_peered":"2024-05-26T12:00:28.571502+0000","last_unstale":"2024-05-26T12:01:07.946543+0000","last_undegraded":"2024-05-26T12:01:07.946543+0000","last_fullsized":"2024-05-26T12:01:07.946543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:08:01.040087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939937+0000","last_change":"2024-05-26T12:00:32.616575+0000","last_active":"2024-05-26T12:01:07.939937+0000","last_peered":"2024-05-26T12:01:07.939937+0000","last_clean":"2024-05-26T12:01:07.939937+0000","last_became_active":"2024-05-26T12:00:32.616261+0000","last_became_peered":"2024-05-26T12:00:32.616261+0000","last_unstale":"2024-05-26T12:01:07.939937+0000","last_undegraded":"2024-05-26T12:01:07.939937+0000","last_fullsized":"2024-05-26T12:01:07.939937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:06:50.012838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945052+0000","last_change":"2024-05-26T12:00:28.584129+0000","last_active":"2024-05-26T12:01:07.945052+0000","last_peered":"2024-05-26T12:01:07.945052+0000","last_clean":"2024-05-26T12:01:07.945052+0000","last_became_active":"2024-05-26T12:00:28.583046+0000","last_became_peered":"2024-05-26T12:00:28.583046+0000","last_unstale":"2024-05-26T12:01:07.945052+0000","last_undegraded":"2024-05-26T12:01:07.945052+0000","last_fullsized":"2024-05-26T12:01:07.945052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:47.635696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941649+0000","last_change":"2024-05-26T12:00:32.610148+0000","last_active":"2024-05-26T12:01:07.941649+0000","last_peered":"2024-05-26T12:01:07.941649+0000","last_clean":"2024-05-26T12:01:07.941649+0000","last_became_active":"2024-05-26T12:00:32.609849+0000","last_became_peered":"2024-05-26T12:00:32.609849+0000","last_unstale":"2024-05-26T12:01:07.941649+0000","last_undegraded":"2024-05-26T12:01:07.941649+0000","last_fullsized":"2024-05-26T12:01:07.941649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:37:08.938164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":78,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:15.543771+0000","last_change":"2024-05-26T12:00:28.578831+0000","last_active":"2024-05-26T12:01:15.543771+0000","last_peered":"2024-05-26T12:01:15.543771+0000","last_clean":"2024-05-26T12:01:15.543771+0000","last_became_active":"2024-05-26T12:00:28.578484+0000","last_became_peered":"2024-05-26T12:00:28.578484+0000","last_unstale":"2024-05-26T12:01:15.543771+0000","last_undegraded":"2024-05-26T12:01:15.543771+0000","last_fullsized":"2024-05-26T12:01:15.543771+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:11:07.127948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947703+0000","last_change":"2024-05-26T12:00:32.618151+0000","last_active":"2024-05-26T12:01:07.947703+0000","last_peered":"2024-05-26T12:01:07.947703+0000","last_clean":"2024-05-26T12:01:07.947703+0000","last_became_active":"2024-05-26T12:00:32.616835+0000","last_became_peered":"2024-05-26T12:00:32.616835+0000","last_unstale":"2024-05-26T12:01:07.947703+0000","last_undegraded":"2024-05-26T12:01:07.947703+0000","last_fullsized":"2024-05-26T12:01:07.947703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:17:57.682573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944986+0000","last_change":"2024-05-26T12:00:28.579624+0000","last_active":"2024-05-26T12:01:07.944986+0000","last_peered":"2024-05-26T12:01:07.944986+0000","last_clean":"2024-05-26T12:01:07.944986+0000","last_became_active":"2024-05-26T12:00:28.578691+0000","last_became_peered":"2024-05-26T12:00:28.578691+0000","last_unstale":"2024-05-26T12:01:07.944986+0000","last_undegraded":"2024-05-26T12:01:07.944986+0000","last_fullsized":"2024-05-26T12:01:07.944986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:38:12.319875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941402+0000","last_change":"2024-05-26T12:00:32.610573+0000","last_active":"2024-05-26T12:01:07.941402+0000","last_peered":"2024-05-26T12:01:07.941402+0000","last_clean":"2024-05-26T12:01:07.941402+0000","last_became_active":"2024-05-26T12:00:32.609824+0000","last_became_peered":"2024-05-26T12:00:32.609824+0000","last_unstale":"2024-05-26T12:01:07.941402+0000","last_undegraded":"2024-05-26T12:01:07.941402+0000","last_fullsized":"2024-05-26T12:01:07.941402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:56.631874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945545+0000","last_change":"2024-05-26T12:00:28.672441+0000","last_active":"2024-05-26T12:01:07.945545+0000","last_peered":"2024-05-26T12:01:07.945545+0000","last_clean":"2024-05-26T12:01:07.945545+0000","last_became_active":"2024-05-26T12:00:28.672039+0000","last_became_peered":"2024-05-26T12:00:28.672039+0000","last_unstale":"2024-05-26T12:01:07.945545+0000","last_undegraded":"2024-05-26T12:01:07.945545+0000","last_fullsized":"2024-05-26T12:01:07.945545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:14:09.757458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947488+0000","last_change":"2024-05-26T12:00:32.612289+0000","last_active":"2024-05-26T12:01:07.947488+0000","last_peered":"2024-05-26T12:01:07.947488+0000","last_clean":"2024-05-26T12:01:07.947488+0000","last_became_active":"2024-05-26T12:00:32.611930+0000","last_became_peered":"2024-05-26T12:00:32.611930+0000","last_unstale":"2024-05-26T12:01:07.947488+0000","last_undegraded":"2024-05-26T12:01:07.947488+0000","last_fullsized":"2024-05-26T12:01:07.947488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:05:55.351242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'30","reported_seq":130,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:15.491367+0000","last_change":"2024-05-26T12:01:08.493089+0000","last_active":"2024-05-26T12:01:15.491367+0000","last_peered":"2024-05-26T12:01:15.491367+0000","last_clean":"2024-05-26T12:01:15.491367+0000","last_became_active":"2024-05-26T12:01:08.471966+0000","last_became_peered":"2024-05-26T12:01:08.471966+0000","last_unstale":"2024-05-26T12:01:15.491367+0000","last_undegraded":"2024-05-26T12:01:15.491367+0000","last_fullsized":"2024-05-26T12:01:15.491367+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:51.413474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":55505,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":47,"num_read_kb":71,"num_write":32,"num_write_kb":64,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":28083,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946125+0000","last_change":"2024-05-26T12:00:32.614557+0000","last_active":"2024-05-26T12:01:07.946125+0000","last_peered":"2024-05-26T12:01:07.946125+0000","last_clean":"2024-05-26T12:01:07.946125+0000","last_became_active":"2024-05-26T12:00:32.613806+0000","last_became_peered":"2024-05-26T12:00:32.613806+0000","last_unstale":"2024-05-26T12:01:07.946125+0000","last_undegraded":"2024-05-26T12:01:07.946125+0000","last_fullsized":"2024-05-26T12:01:07.946125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:11:11.279479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948649+0000","last_change":"2024-05-26T12:00:28.577551+0000","last_active":"2024-05-26T12:01:07.948649+0000","last_peered":"2024-05-26T12:01:07.948649+0000","last_clean":"2024-05-26T12:01:07.948649+0000","last_became_active":"2024-05-26T12:00:28.577203+0000","last_became_peered":"2024-05-26T12:00:28.577203+0000","last_unstale":"2024-05-26T12:01:07.948649+0000","last_undegraded":"2024-05-26T12:01:07.948649+0000","last_fullsized":"2024-05-26T12:01:07.948649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:22:54.906432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946715+0000","last_change":"2024-05-26T12:00:32.612029+0000","last_active":"2024-05-26T12:01:07.946715+0000","last_peered":"2024-05-26T12:01:07.946715+0000","last_clean":"2024-05-26T12:01:07.946715+0000","last_became_active":"2024-05-26T12:00:32.611660+0000","last_became_peered":"2024-05-26T12:00:32.611660+0000","last_unstale":"2024-05-26T12:01:07.946715+0000","last_undegraded":"2024-05-26T12:01:07.946715+0000","last_fullsized":"2024-05-26T12:01:07.946715+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:15:27.278693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.960692+0000","last_change":"2024-05-26T12:01:07.960293+0000","last_active":"2024-05-26T12:01:07.960692+0000","last_peered":"2024-05-26T12:01:07.960692+0000","last_clean":"2024-05-26T12:01:07.960692+0000","last_became_active":"2024-05-26T12:01:07.947178+0000","last_became_peered":"2024-05-26T12:01:07.947178+0000","last_unstale":"2024-05-26T12:01:07.960692+0000","last_undegraded":"2024-05-26T12:01:07.960692+0000","last_fullsized":"2024-05-26T12:01:07.960692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:28:23.807776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948672+0000","last_change":"2024-05-26T12:01:07.948672+0000","last_active":"2024-05-26T12:01:07.948672+0000","last_peered":"2024-05-26T12:01:07.948672+0000","last_clean":"2024-05-26T12:01:07.948672+0000","last_became_active":"2024-05-26T12:01:07.948285+0000","last_became_peered":"2024-05-26T12:01:07.948285+0000","last_unstale":"2024-05-26T12:01:07.948672+0000","last_undegraded":"2024-05-26T12:01:07.948672+0000","last_fullsized":"2024-05-26T12:01:07.948672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:49:39.764352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:13.541770+0000","last_change":"2024-05-26T12:00:28.578507+0000","last_active":"2024-05-26T12:01:13.541770+0000","last_peered":"2024-05-26T12:01:13.541770+0000","last_clean":"2024-05-26T12:01:13.541770+0000","last_became_active":"2024-05-26T12:00:28.578131+0000","last_became_peered":"2024-05-26T12:00:28.578131+0000","last_unstale":"2024-05-26T12:01:13.541770+0000","last_undegraded":"2024-05-26T12:01:13.541770+0000","last_fullsized":"2024-05-26T12:01:13.541770+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:14:52.015296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.943799+0000","last_change":"2024-05-26T12:00:32.618061+0000","last_active":"2024-05-26T12:01:07.943799+0000","last_peered":"2024-05-26T12:01:07.943799+0000","last_clean":"2024-05-26T12:01:07.943799+0000","last_became_active":"2024-05-26T12:00:32.617320+0000","last_became_peered":"2024-05-26T12:00:32.617320+0000","last_unstale":"2024-05-26T12:01:07.943799+0000","last_undegraded":"2024-05-26T12:01:07.943799+0000","last_fullsized":"2024-05-26T12:01:07.943799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:57:33.972120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945355+0000","last_change":"2024-05-26T12:00:28.672890+0000","last_active":"2024-05-26T12:01:07.945355+0000","last_peered":"2024-05-26T12:01:07.945355+0000","last_clean":"2024-05-26T12:01:07.945355+0000","last_became_active":"2024-05-26T12:00:28.672523+0000","last_became_peered":"2024-05-26T12:00:28.672523+0000","last_unstale":"2024-05-26T12:01:07.945355+0000","last_undegraded":"2024-05-26T12:01:07.945355+0000","last_fullsized":"2024-05-26T12:01:07.945355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:46:00.281960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945104+0000","last_change":"2024-05-26T12:00:32.618020+0000","last_active":"2024-05-26T12:01:07.945104+0000","last_peered":"2024-05-26T12:01:07.945104+0000","last_clean":"2024-05-26T12:01:07.945104+0000","last_became_active":"2024-05-26T12:00:32.617147+0000","last_became_peered":"2024-05-26T12:00:32.617147+0000","last_unstale":"2024-05-26T12:01:07.945104+0000","last_undegraded":"2024-05-26T12:01:07.945104+0000","last_fullsized":"2024-05-26T12:01:07.945104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:54:06.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945919+0000","last_change":"2024-05-26T12:00:28.573265+0000","last_active":"2024-05-26T12:01:07.945919+0000","last_peered":"2024-05-26T12:01:07.945919+0000","last_clean":"2024-05-26T12:01:07.945919+0000","last_became_active":"2024-05-26T12:00:28.572806+0000","last_became_peered":"2024-05-26T12:00:28.572806+0000","last_unstale":"2024-05-26T12:01:07.945919+0000","last_undegraded":"2024-05-26T12:01:07.945919+0000","last_fullsized":"2024-05-26T12:01:07.945919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:20:11.192942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470842+0000","last_change":"2024-05-26T12:01:08.470842+0000","last_active":"2024-05-26T12:01:08.470842+0000","last_peered":"2024-05-26T12:01:08.470842+0000","last_clean":"2024-05-26T12:01:08.470842+0000","last_became_active":"2024-05-26T12:01:08.470548+0000","last_became_peered":"2024-05-26T12:01:08.470548+0000","last_unstale":"2024-05-26T12:01:08.470842+0000","last_undegraded":"2024-05-26T12:01:08.470842+0000","last_fullsized":"2024-05-26T12:01:08.470842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:06:11.284570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941944+0000","last_change":"2024-05-26T12:00:28.672972+0000","last_active":"2024-05-26T12:01:07.941944+0000","last_peered":"2024-05-26T12:01:07.941944+0000","last_clean":"2024-05-26T12:01:07.941944+0000","last_became_active":"2024-05-26T12:00:28.672626+0000","last_became_peered":"2024-05-26T12:00:28.672626+0000","last_unstale":"2024-05-26T12:01:07.941944+0000","last_undegraded":"2024-05-26T12:01:07.941944+0000","last_fullsized":"2024-05-26T12:01:07.941944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:34:31.449986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944509+0000","last_change":"2024-05-26T12:00:32.622325+0000","last_active":"2024-05-26T12:01:07.944509+0000","last_peered":"2024-05-26T12:01:07.944509+0000","last_clean":"2024-05-26T12:01:07.944509+0000","last_became_active":"2024-05-26T12:00:32.621970+0000","last_became_peered":"2024-05-26T12:00:32.621970+0000","last_unstale":"2024-05-26T12:01:07.944509+0000","last_undegraded":"2024-05-26T12:01:07.944509+0000","last_fullsized":"2024-05-26T12:01:07.944509+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:32:36.252057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:15.543822+0000","last_change":"2024-05-26T12:00:28.580235+0000","last_active":"2024-05-26T12:01:15.543822+0000","last_peered":"2024-05-26T12:01:15.543822+0000","last_clean":"2024-05-26T12:01:15.543822+0000","last_became_active":"2024-05-26T12:00:28.579812+0000","last_became_peered":"2024-05-26T12:00:28.579812+0000","last_unstale":"2024-05-26T12:01:15.543822+0000","last_undegraded":"2024-05-26T12:01:15.543822+0000","last_fullsized":"2024-05-26T12:01:15.543822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:10:06.107316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944229+0000","last_change":"2024-05-26T12:00:32.625187+0000","last_active":"2024-05-26T12:01:07.944229+0000","last_peered":"2024-05-26T12:01:07.944229+0000","last_clean":"2024-05-26T12:01:07.944229+0000","last_became_active":"2024-05-26T12:00:32.624833+0000","last_became_peered":"2024-05-26T12:00:32.624833+0000","last_unstale":"2024-05-26T12:01:07.944229+0000","last_undegraded":"2024-05-26T12:01:07.944229+0000","last_fullsized":"2024-05-26T12:01:07.944229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:49:06.502017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.950791+0000","last_change":"2024-05-26T12:01:07.950791+0000","last_active":"2024-05-26T12:01:07.950791+0000","last_peered":"2024-05-26T12:01:07.950791+0000","last_clean":"2024-05-26T12:01:07.950791+0000","last_became_active":"2024-05-26T12:01:07.950397+0000","last_became_peered":"2024-05-26T12:01:07.950397+0000","last_unstale":"2024-05-26T12:01:07.950791+0000","last_undegraded":"2024-05-26T12:01:07.950791+0000","last_fullsized":"2024-05-26T12:01:07.950791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:20:15.193328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:14.542533+0000","last_change":"2024-05-26T12:00:28.671869+0000","last_active":"2024-05-26T12:01:14.542533+0000","last_peered":"2024-05-26T12:01:14.542533+0000","last_clean":"2024-05-26T12:01:14.542533+0000","last_became_active":"2024-05-26T12:00:28.671529+0000","last_became_peered":"2024-05-26T12:00:28.671529+0000","last_unstale":"2024-05-26T12:01:14.542533+0000","last_undegraded":"2024-05-26T12:01:14.542533+0000","last_fullsized":"2024-05-26T12:01:14.542533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:17.224354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940175+0000","last_change":"2024-05-26T12:00:32.621164+0000","last_active":"2024-05-26T12:01:07.940175+0000","last_peered":"2024-05-26T12:01:07.940175+0000","last_clean":"2024-05-26T12:01:07.940175+0000","last_became_active":"2024-05-26T12:00:32.618422+0000","last_became_peered":"2024-05-26T12:00:32.618422+0000","last_unstale":"2024-05-26T12:01:07.940175+0000","last_undegraded":"2024-05-26T12:01:07.940175+0000","last_fullsized":"2024-05-26T12:01:07.940175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:33:29.567862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470099+0000","last_change":"2024-05-26T12:00:28.579949+0000","last_active":"2024-05-26T12:01:08.470099+0000","last_peered":"2024-05-26T12:01:08.470099+0000","last_clean":"2024-05-26T12:01:08.470099+0000","last_became_active":"2024-05-26T12:00:28.579587+0000","last_became_peered":"2024-05-26T12:00:28.579587+0000","last_unstale":"2024-05-26T12:01:08.470099+0000","last_undegraded":"2024-05-26T12:01:08.470099+0000","last_fullsized":"2024-05-26T12:01:08.470099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:26:18.491139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944176+0000","last_change":"2024-05-26T11:59:02.705869+0000","last_active":"2024-05-26T12:01:07.944176+0000","last_peered":"2024-05-26T12:01:07.944176+0000","last_clean":"2024-05-26T12:01:07.944176+0000","last_became_active":"2024-05-26T11:59:02.700918+0000","last_became_peered":"2024-05-26T11:59:02.700918+0000","last_unstale":"2024-05-26T12:01:07.944176+0000","last_undegraded":"2024-05-26T12:01:07.944176+0000","last_fullsized":"2024-05-26T12:01:07.944176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_clean_scrub_stamp":"2024-05-26T11:55:35.143931+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:01:24.735040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946403+0000","last_change":"2024-05-26T12:00:32.606843+0000","last_active":"2024-05-26T12:01:07.946403+0000","last_peered":"2024-05-26T12:01:07.946403+0000","last_clean":"2024-05-26T12:01:07.946403+0000","last_became_active":"2024-05-26T12:00:32.606552+0000","last_became_peered":"2024-05-26T12:00:32.606552+0000","last_unstale":"2024-05-26T12:01:07.946403+0000","last_undegraded":"2024-05-26T12:01:07.946403+0000","last_fullsized":"2024-05-26T12:01:07.946403+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:43:03.113199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944060+0000","last_change":"2024-05-26T12:00:28.581819+0000","last_active":"2024-05-26T12:01:07.944060+0000","last_peered":"2024-05-26T12:01:07.944060+0000","last_clean":"2024-05-26T12:01:07.944060+0000","last_became_active":"2024-05-26T12:00:28.581430+0000","last_became_peered":"2024-05-26T12:00:28.581430+0000","last_unstale":"2024-05-26T12:01:07.944060+0000","last_undegraded":"2024-05-26T12:01:07.944060+0000","last_fullsized":"2024-05-26T12:01:07.944060+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:13:42.556238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946242+0000","last_change":"2024-05-26T12:00:28.672396+0000","last_active":"2024-05-26T12:01:07.946242+0000","last_peered":"2024-05-26T12:01:07.946242+0000","last_clean":"2024-05-26T12:01:07.946242+0000","last_became_active":"2024-05-26T12:00:28.672045+0000","last_became_peered":"2024-05-26T12:00:28.672045+0000","last_unstale":"2024-05-26T12:01:07.946242+0000","last_undegraded":"2024-05-26T12:01:07.946242+0000","last_fullsized":"2024-05-26T12:01:07.946242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:02:25.925210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946831+0000","last_change":"2024-05-26T12:00:32.606876+0000","last_active":"2024-05-26T12:01:07.946831+0000","last_peered":"2024-05-26T12:01:07.946831+0000","last_clean":"2024-05-26T12:01:07.946831+0000","last_became_active":"2024-05-26T12:00:32.606161+0000","last_became_peered":"2024-05-26T12:00:32.606161+0000","last_unstale":"2024-05-26T12:01:07.946831+0000","last_undegraded":"2024-05-26T12:01:07.946831+0000","last_fullsized":"2024-05-26T12:01:07.946831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:47:02.451447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'13","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:09.624746+0000","last_change":"2024-05-26T12:01:07.955816+0000","last_active":"2024-05-26T12:01:09.624746+0000","last_peered":"2024-05-26T12:01:09.624746+0000","last_clean":"2024-05-26T12:01:09.624746+0000","last_became_active":"2024-05-26T12:01:07.947443+0000","last_became_peered":"2024-05-26T12:01:07.947443+0000","last_unstale":"2024-05-26T12:01:09.624746+0000","last_undegraded":"2024-05-26T12:01:09.624746+0000","last_fullsized":"2024-05-26T12:01:09.624746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:30:00.005339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940312+0000","last_change":"2024-05-26T12:00:32.616713+0000","last_active":"2024-05-26T12:01:07.940312+0000","last_peered":"2024-05-26T12:01:07.940312+0000","last_clean":"2024-05-26T12:01:07.940312+0000","last_became_active":"2024-05-26T12:00:32.616382+0000","last_became_peered":"2024-05-26T12:00:32.616382+0000","last_unstale":"2024-05-26T12:01:07.940312+0000","last_undegraded":"2024-05-26T12:01:07.940312+0000","last_fullsized":"2024-05-26T12:01:07.940312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:12:42.795794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470352+0000","last_change":"2024-05-26T12:00:28.971347+0000","last_active":"2024-05-26T12:01:08.470352+0000","last_peered":"2024-05-26T12:01:08.470352+0000","last_clean":"2024-05-26T12:01:08.470352+0000","last_became_active":"2024-05-26T12:00:28.970911+0000","last_became_peered":"2024-05-26T12:00:28.970911+0000","last_unstale":"2024-05-26T12:01:08.470352+0000","last_undegraded":"2024-05-26T12:01:08.470352+0000","last_fullsized":"2024-05-26T12:01:08.470352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:47:45.306875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940960+0000","last_change":"2024-05-26T12:00:32.614238+0000","last_active":"2024-05-26T12:01:07.940960+0000","last_peered":"2024-05-26T12:01:07.940960+0000","last_clean":"2024-05-26T12:01:07.940960+0000","last_became_active":"2024-05-26T12:00:32.607018+0000","last_became_peered":"2024-05-26T12:00:32.607018+0000","last_unstale":"2024-05-26T12:01:07.940960+0000","last_undegraded":"2024-05-26T12:01:07.940960+0000","last_fullsized":"2024-05-26T12:01:07.940960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:31:06.250615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947096+0000","last_change":"2024-05-26T12:00:28.671961+0000","last_active":"2024-05-26T12:01:07.947096+0000","last_peered":"2024-05-26T12:01:07.947096+0000","last_clean":"2024-05-26T12:01:07.947096+0000","last_became_active":"2024-05-26T12:00:28.671672+0000","last_became_peered":"2024-05-26T12:00:28.671672+0000","last_unstale":"2024-05-26T12:01:07.947096+0000","last_undegraded":"2024-05-26T12:01:07.947096+0000","last_fullsized":"2024-05-26T12:01:07.947096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:05.646044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945323+0000","last_change":"2024-05-26T12:00:32.611432+0000","last_active":"2024-05-26T12:01:07.945323+0000","last_peered":"2024-05-26T12:01:07.945323+0000","last_clean":"2024-05-26T12:01:07.945323+0000","last_became_active":"2024-05-26T12:00:32.610686+0000","last_became_peered":"2024-05-26T12:00:32.610686+0000","last_unstale":"2024-05-26T12:01:07.945323+0000","last_undegraded":"2024-05-26T12:01:07.945323+0000","last_fullsized":"2024-05-26T12:01:07.945323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:52:36.459552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939974+0000","last_change":"2024-05-26T12:00:28.576723+0000","last_active":"2024-05-26T12:01:07.939974+0000","last_peered":"2024-05-26T12:01:07.939974+0000","last_clean":"2024-05-26T12:01:07.939974+0000","last_became_active":"2024-05-26T12:00:28.576380+0000","last_became_peered":"2024-05-26T12:00:28.576380+0000","last_unstale":"2024-05-26T12:01:07.939974+0000","last_undegraded":"2024-05-26T12:01:07.939974+0000","last_fullsized":"2024-05-26T12:01:07.939974+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:49:25.611830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942224+0000","last_change":"2024-05-26T12:00:32.621124+0000","last_active":"2024-05-26T12:01:07.942224+0000","last_peered":"2024-05-26T12:01:07.942224+0000","last_clean":"2024-05-26T12:01:07.942224+0000","last_became_active":"2024-05-26T12:00:32.620740+0000","last_became_peered":"2024-05-26T12:00:32.620740+0000","last_unstale":"2024-05-26T12:01:07.942224+0000","last_undegraded":"2024-05-26T12:01:07.942224+0000","last_fullsized":"2024-05-26T12:01:07.942224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:15:16.123821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948705+0000","last_change":"2024-05-26T12:00:28.572222+0000","last_active":"2024-05-26T12:01:07.948705+0000","last_peered":"2024-05-26T12:01:07.948705+0000","last_clean":"2024-05-26T12:01:07.948705+0000","last_became_active":"2024-05-26T12:00:28.571459+0000","last_became_peered":"2024-05-26T12:00:28.571459+0000","last_unstale":"2024-05-26T12:01:07.948705+0000","last_undegraded":"2024-05-26T12:01:07.948705+0000","last_fullsized":"2024-05-26T12:01:07.948705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:49:56.234550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940179+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:01:07.940179+0000","last_peered":"2024-05-26T12:01:07.940179+0000","last_clean":"2024-05-26T12:01:07.940179+0000","last_became_active":"2024-05-26T12:00:32.617579+0000","last_became_peered":"2024-05-26T12:00:32.617579+0000","last_unstale":"2024-05-26T12:01:07.940179+0000","last_undegraded":"2024-05-26T12:01:07.940179+0000","last_fullsized":"2024-05-26T12:01:07.940179+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:46:57.045155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945531+0000","last_change":"2024-05-26T12:00:28.584555+0000","last_active":"2024-05-26T12:01:07.945531+0000","last_peered":"2024-05-26T12:01:07.945531+0000","last_clean":"2024-05-26T12:01:07.945531+0000","last_became_active":"2024-05-26T12:00:28.583750+0000","last_became_peered":"2024-05-26T12:00:28.583750+0000","last_unstale":"2024-05-26T12:01:07.945531+0000","last_undegraded":"2024-05-26T12:01:07.945531+0000","last_fullsized":"2024-05-26T12:01:07.945531+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:26:50.312369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944253+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:01:07.944253+0000","last_peered":"2024-05-26T12:01:07.944253+0000","last_clean":"2024-05-26T12:01:07.944253+0000","last_became_active":"2024-05-26T12:00:32.615758+0000","last_became_peered":"2024-05-26T12:00:32.615758+0000","last_unstale":"2024-05-26T12:01:07.944253+0000","last_undegraded":"2024-05-26T12:01:07.944253+0000","last_fullsized":"2024-05-26T12:01:07.944253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:17:52.776655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:12.540307+0000","last_change":"2024-05-26T12:00:28.971186+0000","last_active":"2024-05-26T12:01:12.540307+0000","last_peered":"2024-05-26T12:01:12.540307+0000","last_clean":"2024-05-26T12:01:12.540307+0000","last_became_active":"2024-05-26T12:00:28.970706+0000","last_became_peered":"2024-05-26T12:00:28.970706+0000","last_unstale":"2024-05-26T12:01:12.540307+0000","last_undegraded":"2024-05-26T12:01:12.540307+0000","last_fullsized":"2024-05-26T12:01:12.540307+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:39:12.139262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":24,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945309+0000","last_change":"2024-05-26T12:00:32.623244+0000","last_active":"2024-05-26T12:01:07.945309+0000","last_peered":"2024-05-26T12:01:07.945309+0000","last_clean":"2024-05-26T12:01:07.945309+0000","last_became_active":"2024-05-26T12:00:32.622057+0000","last_became_peered":"2024-05-26T12:00:32.622057+0000","last_unstale":"2024-05-26T12:01:07.945309+0000","last_undegraded":"2024-05-26T12:01:07.945309+0000","last_fullsized":"2024-05-26T12:01:07.945309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:37:09.871626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945442+0000","last_change":"2024-05-26T12:00:28.576115+0000","last_active":"2024-05-26T12:01:07.945442+0000","last_peered":"2024-05-26T12:01:07.945442+0000","last_clean":"2024-05-26T12:01:07.945442+0000","last_became_active":"2024-05-26T12:00:28.575704+0000","last_became_peered":"2024-05-26T12:00:28.575704+0000","last_unstale":"2024-05-26T12:01:07.945442+0000","last_undegraded":"2024-05-26T12:01:07.945442+0000","last_fullsized":"2024-05-26T12:01:07.945442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:28:05.142390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944558+0000","last_change":"2024-05-26T12:00:32.611997+0000","last_active":"2024-05-26T12:01:07.944558+0000","last_peered":"2024-05-26T12:01:07.944558+0000","last_clean":"2024-05-26T12:01:07.944558+0000","last_became_active":"2024-05-26T12:00:32.611646+0000","last_became_peered":"2024-05-26T12:00:32.611646+0000","last_unstale":"2024-05-26T12:01:07.944558+0000","last_undegraded":"2024-05-26T12:01:07.944558+0000","last_fullsized":"2024-05-26T12:01:07.944558+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:50:15.856048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946931+0000","last_change":"2024-05-26T12:00:28.576500+0000","last_active":"2024-05-26T12:01:07.946931+0000","last_peered":"2024-05-26T12:01:07.946931+0000","last_clean":"2024-05-26T12:01:07.946931+0000","last_became_active":"2024-05-26T12:00:28.576171+0000","last_became_peered":"2024-05-26T12:00:28.576171+0000","last_unstale":"2024-05-26T12:01:07.946931+0000","last_undegraded":"2024-05-26T12:01:07.946931+0000","last_fullsized":"2024-05-26T12:01:07.946931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:44:54.786997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944854+0000","last_change":"2024-05-26T12:00:32.617550+0000","last_active":"2024-05-26T12:01:07.944854+0000","last_peered":"2024-05-26T12:01:07.944854+0000","last_clean":"2024-05-26T12:01:07.944854+0000","last_became_active":"2024-05-26T12:00:32.616821+0000","last_became_peered":"2024-05-26T12:00:32.616821+0000","last_unstale":"2024-05-26T12:01:07.944854+0000","last_undegraded":"2024-05-26T12:01:07.944854+0000","last_fullsized":"2024-05-26T12:01:07.944854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:49:59.956694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942311+0000","last_change":"2024-05-26T12:00:32.623314+0000","last_active":"2024-05-26T12:01:07.942311+0000","last_peered":"2024-05-26T12:01:07.942311+0000","last_clean":"2024-05-26T12:01:07.942311+0000","last_became_active":"2024-05-26T12:00:32.622187+0000","last_became_peered":"2024-05-26T12:00:32.622187+0000","last_unstale":"2024-05-26T12:01:07.942311+0000","last_undegraded":"2024-05-26T12:01:07.942311+0000","last_fullsized":"2024-05-26T12:01:07.942311+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:19:08.026803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948805+0000","last_change":"2024-05-26T12:00:32.614814+0000","last_active":"2024-05-26T12:01:07.948805+0000","last_peered":"2024-05-26T12:01:07.948805+0000","last_clean":"2024-05-26T12:01:07.948805+0000","last_became_active":"2024-05-26T12:00:32.614439+0000","last_became_peered":"2024-05-26T12:00:32.614439+0000","last_unstale":"2024-05-26T12:01:07.948805+0000","last_undegraded":"2024-05-26T12:01:07.948805+0000","last_fullsized":"2024-05-26T12:01:07.948805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:56:10.817914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948574+0000","last_change":"2024-05-26T12:00:32.610170+0000","last_active":"2024-05-26T12:01:07.948574+0000","last_peered":"2024-05-26T12:01:07.948574+0000","last_clean":"2024-05-26T12:01:07.948574+0000","last_became_active":"2024-05-26T12:00:32.609783+0000","last_became_peered":"2024-05-26T12:00:32.609783+0000","last_unstale":"2024-05-26T12:01:07.948574+0000","last_undegraded":"2024-05-26T12:01:07.948574+0000","last_fullsized":"2024-05-26T12:01:07.948574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:33:52.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941107+0000","last_change":"2024-05-26T12:00:32.609819+0000","last_active":"2024-05-26T12:01:07.941107+0000","last_peered":"2024-05-26T12:01:07.941107+0000","last_clean":"2024-05-26T12:01:07.941107+0000","last_became_active":"2024-05-26T12:00:32.609016+0000","last_became_peered":"2024-05-26T12:00:32.609016+0000","last_unstale":"2024-05-26T12:01:07.941107+0000","last_undegraded":"2024-05-26T12:01:07.941107+0000","last_fullsized":"2024-05-26T12:01:07.941107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:10:09.582603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946791+0000","last_change":"2024-05-26T12:00:32.608904+0000","last_active":"2024-05-26T12:01:07.946791+0000","last_peered":"2024-05-26T12:01:07.946791+0000","last_clean":"2024-05-26T12:01:07.946791+0000","last_became_active":"2024-05-26T12:00:32.608505+0000","last_became_peered":"2024-05-26T12:00:32.608505+0000","last_unstale":"2024-05-26T12:01:07.946791+0000","last_undegraded":"2024-05-26T12:01:07.946791+0000","last_fullsized":"2024-05-26T12:01:07.946791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:34:09.931046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942574+0000","last_change":"2024-05-26T12:00:32.618370+0000","last_active":"2024-05-26T12:01:07.942574+0000","last_peered":"2024-05-26T12:01:07.942574+0000","last_clean":"2024-05-26T12:01:07.942574+0000","last_became_active":"2024-05-26T12:00:32.618022+0000","last_became_peered":"2024-05-26T12:00:32.618022+0000","last_unstale":"2024-05-26T12:01:07.942574+0000","last_undegraded":"2024-05-26T12:01:07.942574+0000","last_fullsized":"2024-05-26T12:01:07.942574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:45:12.652460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.949432+0000","last_change":"2024-05-26T12:00:32.612319+0000","last_active":"2024-05-26T12:01:07.949432+0000","last_peered":"2024-05-26T12:01:07.949432+0000","last_clean":"2024-05-26T12:01:07.949432+0000","last_became_active":"2024-05-26T12:00:32.611973+0000","last_became_peered":"2024-05-26T12:00:32.611973+0000","last_unstale":"2024-05-26T12:01:07.949432+0000","last_undegraded":"2024-05-26T12:01:07.949432+0000","last_fullsized":"2024-05-26T12:01:07.949432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:23:48.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940298+0000","last_change":"2024-05-26T12:00:32.613461+0000","last_active":"2024-05-26T12:01:07.940298+0000","last_peered":"2024-05-26T12:01:07.940298+0000","last_clean":"2024-05-26T12:01:07.940298+0000","last_became_active":"2024-05-26T12:00:32.613068+0000","last_became_peered":"2024-05-26T12:00:32.613068+0000","last_unstale":"2024-05-26T12:01:07.940298+0000","last_undegraded":"2024-05-26T12:01:07.940298+0000","last_fullsized":"2024-05-26T12:01:07.940298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:17:59.606664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944957+0000","last_change":"2024-05-26T12:00:32.615583+0000","last_active":"2024-05-26T12:01:07.944957+0000","last_peered":"2024-05-26T12:01:07.944957+0000","last_clean":"2024-05-26T12:01:07.944957+0000","last_became_active":"2024-05-26T12:00:32.611502+0000","last_became_peered":"2024-05-26T12:00:32.611502+0000","last_unstale":"2024-05-26T12:01:07.944957+0000","last_undegraded":"2024-05-26T12:01:07.944957+0000","last_fullsized":"2024-05-26T12:01:07.944957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:26:47.772010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939923+0000","last_change":"2024-05-26T12:00:32.612190+0000","last_active":"2024-05-26T12:01:07.939923+0000","last_peered":"2024-05-26T12:01:07.939923+0000","last_clean":"2024-05-26T12:01:07.939923+0000","last_became_active":"2024-05-26T12:00:32.611843+0000","last_became_peered":"2024-05-26T12:00:32.611843+0000","last_unstale":"2024-05-26T12:01:07.939923+0000","last_undegraded":"2024-05-26T12:01:07.939923+0000","last_fullsized":"2024-05-26T12:01:07.939923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:23:05.244741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948397+0000","last_change":"2024-05-26T12:00:32.620758+0000","last_active":"2024-05-26T12:01:07.948397+0000","last_peered":"2024-05-26T12:01:07.948397+0000","last_clean":"2024-05-26T12:01:07.948397+0000","last_became_active":"2024-05-26T12:00:32.620401+0000","last_became_peered":"2024-05-26T12:00:32.620401+0000","last_unstale":"2024-05-26T12:01:07.948397+0000","last_undegraded":"2024-05-26T12:01:07.948397+0000","last_fullsized":"2024-05-26T12:01:07.948397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:59:18.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946941+0000","last_change":"2024-05-26T12:00:32.615615+0000","last_active":"2024-05-26T12:01:07.946941+0000","last_peered":"2024-05-26T12:01:07.946941+0000","last_clean":"2024-05-26T12:01:07.946941+0000","last_became_active":"2024-05-26T12:00:32.615304+0000","last_became_peered":"2024-05-26T12:00:32.615304+0000","last_unstale":"2024-05-26T12:01:07.946941+0000","last_undegraded":"2024-05-26T12:01:07.946941+0000","last_fullsized":"2024-05-26T12:01:07.946941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:05:22.568910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942360+0000","last_change":"2024-05-26T12:00:32.627209+0000","last_active":"2024-05-26T12:01:07.942360+0000","last_peered":"2024-05-26T12:01:07.942360+0000","last_clean":"2024-05-26T12:01:07.942360+0000","last_became_active":"2024-05-26T12:00:32.626851+0000","last_became_peered":"2024-05-26T12:00:32.626851+0000","last_unstale":"2024-05-26T12:01:07.942360+0000","last_undegraded":"2024-05-26T12:01:07.942360+0000","last_fullsized":"2024-05-26T12:01:07.942360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:05:40.754565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213552,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":62318,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":262,"num_read_kb":194,"num_write":239,"num_write_kb":129,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":28173,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":536576,"data_stored":186954,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":206,"ondisk_log_size":206,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678041,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":884736,"data_stored":429000,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874269,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1075444,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037795,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27904,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717248,"statfs":{"total":95995035648,"available":95966461952,"internally_reserved":0,"allocated":905216,"data_stored":428840,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":429218,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":1021736,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593779,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":428948,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":485366,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429896,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411399,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":429690,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149762,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":428382,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953533,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":479113,"data_compressed":10339,"data_compressed_allocated":16384,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790007,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":1020933,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":57648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":56205,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":57241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T12:01:20.540 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-26T12:01:20.784 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:21.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:20 smithi186 ceph-mon[36412]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:21.292 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:20 smithi156 ceph-mon[32025]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:21.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:20 smithi192 ceph-mon[36545]: from='client.14733 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:22.212 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:21 smithi156 ceph-mon[32025]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s; 2.3 KiB/s, 1 keys/s, 1 objects/s recovering 2024-05-26T12:01:22.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:21 smithi192 ceph-mon[36545]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s; 2.3 KiB/s, 1 keys/s, 1 objects/s recovering 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.464+0000 7f54443ef700 1 -- 172.21.15.156:0/1485687056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c102bc0 msgr2=0x7f543c103040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.464+0000 7f54443ef700 1 --2- 172.21.15.156:0/1485687056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c102bc0 0x7f543c103040 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f5438009990 tx=0x7f5438009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 -- 172.21.15.156:0/1485687056 shutdown_connections 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 --2- 172.21.15.156:0/1485687056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c103580 0x7f543c10f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 --2- 172.21.15.156:0/1485687056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c1085c0 0x7f543c1089a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 --2- 172.21.15.156:0/1485687056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c102bc0 0x7f543c103040 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 -- 172.21.15.156:0/1485687056 >> 172.21.15.156:0/1485687056 conn(0x7f543c078a40 msgr2=0x7f543c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:22.466 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 -- 172.21.15.156:0/1485687056 shutdown_connections 2024-05-26T12:01:22.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 -- 172.21.15.156:0/1485687056 wait complete. 2024-05-26T12:01:22.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 Processor -- start 2024-05-26T12:01:22.467 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.465+0000 7f54443ef700 1 -- start start 2024-05-26T12:01:22.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 0x7f543c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:22.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:22.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c1085c0 0x7f543c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:22.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f543c1126e0 con 0x7f543c103580 2024-05-26T12:01:22.468 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f543c112560 con 0x7f543c102bc0 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54443ef700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f543c112860 con 0x7f543c1085c0 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41126/0 (socket says 172.21.15.156:41126) 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 -- 172.21.15.156:0/1829320555 learned_addr learned my addr 172.21.15.156:0/1829320555 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544298c700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c1085c0 0x7f543c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544218b700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 0x7f543c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 -- 172.21.15.156:0/1829320555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c1085c0 msgr2=0x7f543c199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c1085c0 0x7f543c199ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 -- 172.21.15.156:0/1829320555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 msgr2=0x7f543c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 0x7f543c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 -- 172.21.15.156:0/1829320555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f543c19a220 con 0x7f543c103580 2024-05-26T12:01:22.469 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544218b700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 0x7f543c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f544198a700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5438005910 tx=0x7f5438003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.466+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5438027070 con 0x7f543c103580 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.467+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f543c19a4b0 con 0x7f543c103580 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.467+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f543c1a6780 con 0x7f543c103580 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.467+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5438004140 con 0x7f543c103580 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.467+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543800f460 con 0x7f543c103580 2024-05-26T12:01:22.470 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.468+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f543c06a230 con 0x7f543c103580 2024-05-26T12:01:22.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.469+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f543802a070 con 0x7f543c103580 2024-05-26T12:01:22.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.470+0000 7f54337fe700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 0x7f54280b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:22.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.470+0000 7f544218b700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 0x7f54280b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:22.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.470+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f54380e3160 con 0x7f543c103580 2024-05-26T12:01:22.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.470+0000 7f544218b700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 0x7f54280b4e50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f542c005fd0 tx=0x7f542c005de0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:22.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.474+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f5438013070 con 0x7f543c103580 2024-05-26T12:01:22.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:21 smithi186 ceph-mon[36412]: pgmap v318: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 3.3 KiB/s wr, 13 op/s; 2.3 KiB/s, 1 keys/s, 1 objects/s recovering 2024-05-26T12:01:22.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.676+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f543c106260 con 0x7f54280b2990 2024-05-26T12:01:22.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.683+0000 7f54337fe700 1 -- 172.21.15.156:0/1829320555 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258998 (secure 0 0 0) 0x7f543c106260 con 0x7f54280b2990 2024-05-26T12:01:22.687 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:22.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 msgr2=0x7f54280b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:22.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 0x7f54280b4e50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f542c005fd0 tx=0x7f542c005de0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 msgr2=0x7f543c19fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5438005910 tx=0x7f5438003730 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 shutdown_connections 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f54280b2990 0x7f54280b4e50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f543c1085c0 0x7f543c199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f543c102bc0 0x7f543c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 --2- 172.21.15.156:0/1829320555 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f543c103580 0x7f543c19fdf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 >> 172.21.15.156:0/1829320555 conn(0x7f543c078a40 msgr2=0x7f543c100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 shutdown_connections 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:22.686+0000 7f54443ef700 1 -- 172.21.15.156:0/1829320555 wait complete. 2024-05-26T12:01:22.689 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-05-26T12:01:23.498 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2024-05-26T12:01:20.806011+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657177,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":379,"num_read_kb":283,"num_write":369,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2980013,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":355892,"kb_used_data":10876,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585932,"statfs":{"total":1151940427776,"available":1151575994368,"internally_reserved":0,"allocated":11137024,"data_stored":7099890,"data_compressed":182304,"data_compressed_allocated":241664,"data_compressed_original":1990656,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31796,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":134,"num_read_kb":93,"num_write":25,"num_write_kb":40,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":28083,"num_keys_recovered":17,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002943"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942670+0000","last_change":"2024-05-26T12:00:32.620823+0000","last_active":"2024-05-26T12:01:07.942670+0000","last_peered":"2024-05-26T12:01:07.942670+0000","last_clean":"2024-05-26T12:01:07.942670+0000","last_became_active":"2024-05-26T12:00:32.619171+0000","last_became_peered":"2024-05-26T12:00:32.619171+0000","last_unstale":"2024-05-26T12:01:07.942670+0000","last_undegraded":"2024-05-26T12:01:07.942670+0000","last_fullsized":"2024-05-26T12:01:07.942670+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:57:59.459768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940886+0000","last_change":"2024-05-26T12:00:32.618888+0000","last_active":"2024-05-26T12:01:07.940886+0000","last_peered":"2024-05-26T12:01:07.940886+0000","last_clean":"2024-05-26T12:01:07.940886+0000","last_became_active":"2024-05-26T12:00:32.618235+0000","last_became_peered":"2024-05-26T12:00:32.618235+0000","last_unstale":"2024-05-26T12:01:07.940886+0000","last_undegraded":"2024-05-26T12:01:07.940886+0000","last_fullsized":"2024-05-26T12:01:07.940886+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:56:08.852594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940789+0000","last_change":"2024-05-26T12:00:32.613694+0000","last_active":"2024-05-26T12:01:07.940789+0000","last_peered":"2024-05-26T12:01:07.940789+0000","last_clean":"2024-05-26T12:01:07.940789+0000","last_became_active":"2024-05-26T12:00:32.613389+0000","last_became_peered":"2024-05-26T12:00:32.613389+0000","last_unstale":"2024-05-26T12:01:07.940789+0000","last_undegraded":"2024-05-26T12:01:07.940789+0000","last_fullsized":"2024-05-26T12:01:07.940789+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:04.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.471139+0000","last_change":"2024-05-26T12:00:32.620149+0000","last_active":"2024-05-26T12:01:08.471139+0000","last_peered":"2024-05-26T12:01:08.471139+0000","last_clean":"2024-05-26T12:01:08.471139+0000","last_became_active":"2024-05-26T12:00:32.614534+0000","last_became_peered":"2024-05-26T12:00:32.614534+0000","last_unstale":"2024-05-26T12:01:08.471139+0000","last_undegraded":"2024-05-26T12:01:08.471139+0000","last_fullsized":"2024-05-26T12:01:08.471139+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:24:41.144826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945382+0000","last_change":"2024-05-26T12:00:32.605841+0000","last_active":"2024-05-26T12:01:07.945382+0000","last_peered":"2024-05-26T12:01:07.945382+0000","last_clean":"2024-05-26T12:01:07.945382+0000","last_became_active":"2024-05-26T12:00:32.605438+0000","last_became_peered":"2024-05-26T12:00:32.605438+0000","last_unstale":"2024-05-26T12:01:07.945382+0000","last_undegraded":"2024-05-26T12:01:07.945382+0000","last_fullsized":"2024-05-26T12:01:07.945382+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:05:01.341398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944312+0000","last_change":"2024-05-26T12:00:32.625270+0000","last_active":"2024-05-26T12:01:07.944312+0000","last_peered":"2024-05-26T12:01:07.944312+0000","last_clean":"2024-05-26T12:01:07.944312+0000","last_became_active":"2024-05-26T12:00:32.624892+0000","last_became_peered":"2024-05-26T12:00:32.624892+0000","last_unstale":"2024-05-26T12:01:07.944312+0000","last_undegraded":"2024-05-26T12:01:07.944312+0000","last_fullsized":"2024-05-26T12:01:07.944312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:44.581793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.943961+0000","last_change":"2024-05-26T12:00:32.613286+0000","last_active":"2024-05-26T12:01:07.943961+0000","last_peered":"2024-05-26T12:01:07.943961+0000","last_clean":"2024-05-26T12:01:07.943961+0000","last_became_active":"2024-05-26T12:00:32.612896+0000","last_became_peered":"2024-05-26T12:00:32.612896+0000","last_unstale":"2024-05-26T12:01:07.943961+0000","last_undegraded":"2024-05-26T12:01:07.943961+0000","last_fullsized":"2024-05-26T12:01:07.943961+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:34:29.796529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939870+0000","last_change":"2024-05-26T12:00:32.620975+0000","last_active":"2024-05-26T12:01:07.939870+0000","last_peered":"2024-05-26T12:01:07.939870+0000","last_clean":"2024-05-26T12:01:07.939870+0000","last_became_active":"2024-05-26T12:00:32.618576+0000","last_became_peered":"2024-05-26T12:00:32.618576+0000","last_unstale":"2024-05-26T12:01:07.939870+0000","last_undegraded":"2024-05-26T12:01:07.939870+0000","last_fullsized":"2024-05-26T12:01:07.939870+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:51:04.187910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947841+0000","last_change":"2024-05-26T12:00:32.618617+0000","last_active":"2024-05-26T12:01:07.947841+0000","last_peered":"2024-05-26T12:01:07.947841+0000","last_clean":"2024-05-26T12:01:07.947841+0000","last_became_active":"2024-05-26T12:00:32.617354+0000","last_became_peered":"2024-05-26T12:00:32.617354+0000","last_unstale":"2024-05-26T12:01:07.947841+0000","last_undegraded":"2024-05-26T12:01:07.947841+0000","last_fullsized":"2024-05-26T12:01:07.947841+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:04:22.469952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944741+0000","last_change":"2024-05-26T12:00:32.618169+0000","last_active":"2024-05-26T12:01:07.944741+0000","last_peered":"2024-05-26T12:01:07.944741+0000","last_clean":"2024-05-26T12:01:07.944741+0000","last_became_active":"2024-05-26T12:00:32.617709+0000","last_became_peered":"2024-05-26T12:00:32.617709+0000","last_unstale":"2024-05-26T12:01:07.944741+0000","last_undegraded":"2024-05-26T12:01:07.944741+0000","last_fullsized":"2024-05-26T12:01:07.944741+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:21.801512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945015+0000","last_change":"2024-05-26T12:00:32.623233+0000","last_active":"2024-05-26T12:01:07.945015+0000","last_peered":"2024-05-26T12:01:07.945015+0000","last_clean":"2024-05-26T12:01:07.945015+0000","last_became_active":"2024-05-26T12:00:32.622483+0000","last_became_peered":"2024-05-26T12:00:32.622483+0000","last_unstale":"2024-05-26T12:01:07.945015+0000","last_undegraded":"2024-05-26T12:01:07.945015+0000","last_fullsized":"2024-05-26T12:01:07.945015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:14.784275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.471103+0000","last_change":"2024-05-26T12:00:32.614575+0000","last_active":"2024-05-26T12:01:08.471103+0000","last_peered":"2024-05-26T12:01:08.471103+0000","last_clean":"2024-05-26T12:01:08.471103+0000","last_became_active":"2024-05-26T12:00:32.614020+0000","last_became_peered":"2024-05-26T12:00:32.614020+0000","last_unstale":"2024-05-26T12:01:08.471103+0000","last_undegraded":"2024-05-26T12:01:08.471103+0000","last_fullsized":"2024-05-26T12:01:08.471103+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:33:05.747419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946775+0000","last_change":"2024-05-26T12:00:32.616624+0000","last_active":"2024-05-26T12:01:07.946775+0000","last_peered":"2024-05-26T12:01:07.946775+0000","last_clean":"2024-05-26T12:01:07.946775+0000","last_became_active":"2024-05-26T12:00:32.616280+0000","last_became_peered":"2024-05-26T12:00:32.616280+0000","last_unstale":"2024-05-26T12:01:07.946775+0000","last_undegraded":"2024-05-26T12:01:07.946775+0000","last_fullsized":"2024-05-26T12:01:07.946775+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:45:02.466249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940839+0000","last_change":"2024-05-26T12:00:32.617058+0000","last_active":"2024-05-26T12:01:07.940839+0000","last_peered":"2024-05-26T12:01:07.940839+0000","last_clean":"2024-05-26T12:01:07.940839+0000","last_became_active":"2024-05-26T12:00:32.614651+0000","last_became_peered":"2024-05-26T12:00:32.614651+0000","last_unstale":"2024-05-26T12:01:07.940839+0000","last_undegraded":"2024-05-26T12:01:07.940839+0000","last_fullsized":"2024-05-26T12:01:07.940839+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:38:50.234745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.950685+0000","last_change":"2024-05-26T12:01:07.950685+0000","last_active":"2024-05-26T12:01:07.950685+0000","last_peered":"2024-05-26T12:01:07.950685+0000","last_clean":"2024-05-26T12:01:07.950685+0000","last_became_active":"2024-05-26T12:01:07.950383+0000","last_became_peered":"2024-05-26T12:01:07.950383+0000","last_unstale":"2024-05-26T12:01:07.950685+0000","last_undegraded":"2024-05-26T12:01:07.950685+0000","last_fullsized":"2024-05-26T12:01:07.950685+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:02:42.832274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947055+0000","last_change":"2024-05-26T12:00:32.618850+0000","last_active":"2024-05-26T12:01:07.947055+0000","last_peered":"2024-05-26T12:01:07.947055+0000","last_clean":"2024-05-26T12:01:07.947055+0000","last_became_active":"2024-05-26T12:00:32.618494+0000","last_became_peered":"2024-05-26T12:00:32.618494+0000","last_unstale":"2024-05-26T12:01:07.947055+0000","last_undegraded":"2024-05-26T12:01:07.947055+0000","last_fullsized":"2024-05-26T12:01:07.947055+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:23:43.555380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942836+0000","last_change":"2024-05-26T12:00:32.621515+0000","last_active":"2024-05-26T12:01:07.942836+0000","last_peered":"2024-05-26T12:01:07.942836+0000","last_clean":"2024-05-26T12:01:07.942836+0000","last_became_active":"2024-05-26T12:00:32.621091+0000","last_became_peered":"2024-05-26T12:00:32.621091+0000","last_unstale":"2024-05-26T12:01:07.942836+0000","last_undegraded":"2024-05-26T12:01:07.942836+0000","last_fullsized":"2024-05-26T12:01:07.942836+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:32:07.614386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942309+0000","last_change":"2024-05-26T12:00:32.612371+0000","last_active":"2024-05-26T12:01:07.942309+0000","last_peered":"2024-05-26T12:01:07.942309+0000","last_clean":"2024-05-26T12:01:07.942309+0000","last_became_active":"2024-05-26T12:00:32.612082+0000","last_became_peered":"2024-05-26T12:00:32.612082+0000","last_unstale":"2024-05-26T12:01:07.942309+0000","last_undegraded":"2024-05-26T12:01:07.942309+0000","last_fullsized":"2024-05-26T12:01:07.942309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:25:15.850251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.949132+0000","last_change":"2024-05-26T12:00:32.611421+0000","last_active":"2024-05-26T12:01:07.949132+0000","last_peered":"2024-05-26T12:01:07.949132+0000","last_clean":"2024-05-26T12:01:07.949132+0000","last_became_active":"2024-05-26T12:00:32.609716+0000","last_became_peered":"2024-05-26T12:00:32.609716+0000","last_unstale":"2024-05-26T12:01:07.949132+0000","last_undegraded":"2024-05-26T12:01:07.949132+0000","last_fullsized":"2024-05-26T12:01:07.949132+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:55.197945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":67,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:17.549148+0000","last_change":"2024-05-26T12:00:28.578034+0000","last_active":"2024-05-26T12:01:17.549148+0000","last_peered":"2024-05-26T12:01:17.549148+0000","last_clean":"2024-05-26T12:01:17.549148+0000","last_became_active":"2024-05-26T12:00:28.577682+0000","last_became_peered":"2024-05-26T12:00:28.577682+0000","last_unstale":"2024-05-26T12:01:17.549148+0000","last_undegraded":"2024-05-26T12:01:17.549148+0000","last_fullsized":"2024-05-26T12:01:17.549148+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:27:11.135886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":28,"num_read_kb":28,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940258+0000","last_change":"2024-05-26T12:00:32.609857+0000","last_active":"2024-05-26T12:01:07.940258+0000","last_peered":"2024-05-26T12:01:07.940258+0000","last_clean":"2024-05-26T12:01:07.940258+0000","last_became_active":"2024-05-26T12:00:32.609380+0000","last_became_peered":"2024-05-26T12:00:32.609380+0000","last_unstale":"2024-05-26T12:01:07.940258+0000","last_undegraded":"2024-05-26T12:01:07.940258+0000","last_fullsized":"2024-05-26T12:01:07.940258+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:39:04.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940461+0000","last_change":"2024-05-26T12:00:28.671940+0000","last_active":"2024-05-26T12:01:07.940461+0000","last_peered":"2024-05-26T12:01:07.940461+0000","last_clean":"2024-05-26T12:01:07.940461+0000","last_became_active":"2024-05-26T12:00:28.671602+0000","last_became_peered":"2024-05-26T12:00:28.671602+0000","last_unstale":"2024-05-26T12:01:07.940461+0000","last_undegraded":"2024-05-26T12:01:07.940461+0000","last_fullsized":"2024-05-26T12:01:07.940461+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:38:52.069136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941409+0000","last_change":"2024-05-26T12:00:32.613900+0000","last_active":"2024-05-26T12:01:07.941409+0000","last_peered":"2024-05-26T12:01:07.941409+0000","last_clean":"2024-05-26T12:01:07.941409+0000","last_became_active":"2024-05-26T12:00:32.610264+0000","last_became_peered":"2024-05-26T12:00:32.610264+0000","last_unstale":"2024-05-26T12:01:07.941409+0000","last_undegraded":"2024-05-26T12:01:07.941409+0000","last_fullsized":"2024-05-26T12:01:07.941409+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:28:58.712711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.483676+0000","last_change":"2024-05-26T12:01:08.483676+0000","last_active":"2024-05-26T12:01:08.483676+0000","last_peered":"2024-05-26T12:01:08.483676+0000","last_clean":"2024-05-26T12:01:08.483676+0000","last_became_active":"2024-05-26T12:01:08.471437+0000","last_became_peered":"2024-05-26T12:01:08.471437+0000","last_unstale":"2024-05-26T12:01:08.483676+0000","last_undegraded":"2024-05-26T12:01:08.483676+0000","last_fullsized":"2024-05-26T12:01:08.483676+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:43:38.663368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947387+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:01:07.947387+0000","last_peered":"2024-05-26T12:01:07.947387+0000","last_clean":"2024-05-26T12:01:07.947387+0000","last_became_active":"2024-05-26T12:00:32.617702+0000","last_became_peered":"2024-05-26T12:00:32.617702+0000","last_unstale":"2024-05-26T12:01:07.947387+0000","last_undegraded":"2024-05-26T12:01:07.947387+0000","last_fullsized":"2024-05-26T12:01:07.947387+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:25:06.527677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946377+0000","last_change":"2024-05-26T12:00:28.575648+0000","last_active":"2024-05-26T12:01:07.946377+0000","last_peered":"2024-05-26T12:01:07.946377+0000","last_clean":"2024-05-26T12:01:07.946377+0000","last_became_active":"2024-05-26T12:00:28.575170+0000","last_became_peered":"2024-05-26T12:00:28.575170+0000","last_unstale":"2024-05-26T12:01:07.946377+0000","last_undegraded":"2024-05-26T12:01:07.946377+0000","last_fullsized":"2024-05-26T12:01:07.946377+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:09:29.213605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945505+0000","last_change":"2024-05-26T12:00:32.617825+0000","last_active":"2024-05-26T12:01:07.945505+0000","last_peered":"2024-05-26T12:01:07.945505+0000","last_clean":"2024-05-26T12:01:07.945505+0000","last_became_active":"2024-05-26T12:00:32.617517+0000","last_became_peered":"2024-05-26T12:00:32.617517+0000","last_unstale":"2024-05-26T12:01:07.945505+0000","last_undegraded":"2024-05-26T12:01:07.945505+0000","last_fullsized":"2024-05-26T12:01:07.945505+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:36:33.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:17.547509+0000","last_change":"2024-05-26T12:00:28.584639+0000","last_active":"2024-05-26T12:01:17.547509+0000","last_peered":"2024-05-26T12:01:17.547509+0000","last_clean":"2024-05-26T12:01:17.547509+0000","last_became_active":"2024-05-26T12:00:28.583063+0000","last_became_peered":"2024-05-26T12:00:28.583063+0000","last_unstale":"2024-05-26T12:01:17.547509+0000","last_undegraded":"2024-05-26T12:01:17.547509+0000","last_fullsized":"2024-05-26T12:01:17.547509+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:01:25.804547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":29,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946260+0000","last_change":"2024-05-26T12:00:32.609117+0000","last_active":"2024-05-26T12:01:07.946260+0000","last_peered":"2024-05-26T12:01:07.946260+0000","last_clean":"2024-05-26T12:01:07.946260+0000","last_became_active":"2024-05-26T12:00:32.608770+0000","last_became_peered":"2024-05-26T12:00:32.608770+0000","last_unstale":"2024-05-26T12:01:07.946260+0000","last_undegraded":"2024-05-26T12:01:07.946260+0000","last_fullsized":"2024-05-26T12:01:07.946260+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:19:57.422101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":47,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.954880+0000","last_change":"2024-05-26T12:01:07.954880+0000","last_active":"2024-05-26T12:01:07.954880+0000","last_peered":"2024-05-26T12:01:07.954880+0000","last_clean":"2024-05-26T12:01:07.954880+0000","last_became_active":"2024-05-26T12:01:07.949506+0000","last_became_peered":"2024-05-26T12:01:07.949506+0000","last_unstale":"2024-05-26T12:01:07.954880+0000","last_undegraded":"2024-05-26T12:01:07.954880+0000","last_fullsized":"2024-05-26T12:01:07.954880+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:40:37.041619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945386+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:01:07.945386+0000","last_peered":"2024-05-26T12:01:07.945386+0000","last_clean":"2024-05-26T12:01:07.945386+0000","last_became_active":"2024-05-26T12:00:32.615691+0000","last_became_peered":"2024-05-26T12:00:32.615691+0000","last_unstale":"2024-05-26T12:01:07.945386+0000","last_undegraded":"2024-05-26T12:01:07.945386+0000","last_fullsized":"2024-05-26T12:01:07.945386+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:58:26.245149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946543+0000","last_change":"2024-05-26T12:00:28.571893+0000","last_active":"2024-05-26T12:01:07.946543+0000","last_peered":"2024-05-26T12:01:07.946543+0000","last_clean":"2024-05-26T12:01:07.946543+0000","last_became_active":"2024-05-26T12:00:28.571502+0000","last_became_peered":"2024-05-26T12:00:28.571502+0000","last_unstale":"2024-05-26T12:01:07.946543+0000","last_undegraded":"2024-05-26T12:01:07.946543+0000","last_fullsized":"2024-05-26T12:01:07.946543+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:08:01.040087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939937+0000","last_change":"2024-05-26T12:00:32.616575+0000","last_active":"2024-05-26T12:01:07.939937+0000","last_peered":"2024-05-26T12:01:07.939937+0000","last_clean":"2024-05-26T12:01:07.939937+0000","last_became_active":"2024-05-26T12:00:32.616261+0000","last_became_peered":"2024-05-26T12:00:32.616261+0000","last_unstale":"2024-05-26T12:01:07.939937+0000","last_undegraded":"2024-05-26T12:01:07.939937+0000","last_fullsized":"2024-05-26T12:01:07.939937+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:06:50.012838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945052+0000","last_change":"2024-05-26T12:00:28.584129+0000","last_active":"2024-05-26T12:01:07.945052+0000","last_peered":"2024-05-26T12:01:07.945052+0000","last_clean":"2024-05-26T12:01:07.945052+0000","last_became_active":"2024-05-26T12:00:28.583046+0000","last_became_peered":"2024-05-26T12:00:28.583046+0000","last_unstale":"2024-05-26T12:01:07.945052+0000","last_undegraded":"2024-05-26T12:01:07.945052+0000","last_fullsized":"2024-05-26T12:01:07.945052+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:47.635696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941649+0000","last_change":"2024-05-26T12:00:32.610148+0000","last_active":"2024-05-26T12:01:07.941649+0000","last_peered":"2024-05-26T12:01:07.941649+0000","last_clean":"2024-05-26T12:01:07.941649+0000","last_became_active":"2024-05-26T12:00:32.609849+0000","last_became_peered":"2024-05-26T12:00:32.609849+0000","last_unstale":"2024-05-26T12:01:07.941649+0000","last_undegraded":"2024-05-26T12:01:07.941649+0000","last_fullsized":"2024-05-26T12:01:07.941649+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:37:08.938164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":88,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:20.550808+0000","last_change":"2024-05-26T12:00:28.578831+0000","last_active":"2024-05-26T12:01:20.550808+0000","last_peered":"2024-05-26T12:01:20.550808+0000","last_clean":"2024-05-26T12:01:20.550808+0000","last_became_active":"2024-05-26T12:00:28.578484+0000","last_became_peered":"2024-05-26T12:00:28.578484+0000","last_unstale":"2024-05-26T12:01:20.550808+0000","last_undegraded":"2024-05-26T12:01:20.550808+0000","last_fullsized":"2024-05-26T12:01:20.550808+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:11:07.127948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":58,"num_read_kb":58,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947703+0000","last_change":"2024-05-26T12:00:32.618151+0000","last_active":"2024-05-26T12:01:07.947703+0000","last_peered":"2024-05-26T12:01:07.947703+0000","last_clean":"2024-05-26T12:01:07.947703+0000","last_became_active":"2024-05-26T12:00:32.616835+0000","last_became_peered":"2024-05-26T12:00:32.616835+0000","last_unstale":"2024-05-26T12:01:07.947703+0000","last_undegraded":"2024-05-26T12:01:07.947703+0000","last_fullsized":"2024-05-26T12:01:07.947703+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:17:57.682573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944986+0000","last_change":"2024-05-26T12:00:28.579624+0000","last_active":"2024-05-26T12:01:07.944986+0000","last_peered":"2024-05-26T12:01:07.944986+0000","last_clean":"2024-05-26T12:01:07.944986+0000","last_became_active":"2024-05-26T12:00:28.578691+0000","last_became_peered":"2024-05-26T12:00:28.578691+0000","last_unstale":"2024-05-26T12:01:07.944986+0000","last_undegraded":"2024-05-26T12:01:07.944986+0000","last_fullsized":"2024-05-26T12:01:07.944986+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:38:12.319875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941402+0000","last_change":"2024-05-26T12:00:32.610573+0000","last_active":"2024-05-26T12:01:07.941402+0000","last_peered":"2024-05-26T12:01:07.941402+0000","last_clean":"2024-05-26T12:01:07.941402+0000","last_became_active":"2024-05-26T12:00:32.609824+0000","last_became_peered":"2024-05-26T12:00:32.609824+0000","last_unstale":"2024-05-26T12:01:07.941402+0000","last_undegraded":"2024-05-26T12:01:07.941402+0000","last_fullsized":"2024-05-26T12:01:07.941402+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:56.631874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945545+0000","last_change":"2024-05-26T12:00:28.672441+0000","last_active":"2024-05-26T12:01:07.945545+0000","last_peered":"2024-05-26T12:01:07.945545+0000","last_clean":"2024-05-26T12:01:07.945545+0000","last_became_active":"2024-05-26T12:00:28.672039+0000","last_became_peered":"2024-05-26T12:00:28.672039+0000","last_unstale":"2024-05-26T12:01:07.945545+0000","last_undegraded":"2024-05-26T12:01:07.945545+0000","last_fullsized":"2024-05-26T12:01:07.945545+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:14:09.757458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947488+0000","last_change":"2024-05-26T12:00:32.612289+0000","last_active":"2024-05-26T12:01:07.947488+0000","last_peered":"2024-05-26T12:01:07.947488+0000","last_clean":"2024-05-26T12:01:07.947488+0000","last_became_active":"2024-05-26T12:00:32.611930+0000","last_became_peered":"2024-05-26T12:00:32.611930+0000","last_unstale":"2024-05-26T12:01:07.947488+0000","last_undegraded":"2024-05-26T12:01:07.947488+0000","last_fullsized":"2024-05-26T12:01:07.947488+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:05:55.351242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'31","reported_seq":142,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:20.500662+0000","last_change":"2024-05-26T12:01:08.493089+0000","last_active":"2024-05-26T12:01:20.500662+0000","last_peered":"2024-05-26T12:01:20.500662+0000","last_clean":"2024-05-26T12:01:20.500662+0000","last_became_active":"2024-05-26T12:01:08.471966+0000","last_became_peered":"2024-05-26T12:01:08.471966+0000","last_unstale":"2024-05-26T12:01:20.500662+0000","last_undegraded":"2024-05-26T12:01:20.500662+0000","last_fullsized":"2024-05-26T12:01:20.500662+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:51.413474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59762,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":81,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":28083,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946125+0000","last_change":"2024-05-26T12:00:32.614557+0000","last_active":"2024-05-26T12:01:07.946125+0000","last_peered":"2024-05-26T12:01:07.946125+0000","last_clean":"2024-05-26T12:01:07.946125+0000","last_became_active":"2024-05-26T12:00:32.613806+0000","last_became_peered":"2024-05-26T12:00:32.613806+0000","last_unstale":"2024-05-26T12:01:07.946125+0000","last_undegraded":"2024-05-26T12:01:07.946125+0000","last_fullsized":"2024-05-26T12:01:07.946125+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:11:11.279479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948649+0000","last_change":"2024-05-26T12:00:28.577551+0000","last_active":"2024-05-26T12:01:07.948649+0000","last_peered":"2024-05-26T12:01:07.948649+0000","last_clean":"2024-05-26T12:01:07.948649+0000","last_became_active":"2024-05-26T12:00:28.577203+0000","last_became_peered":"2024-05-26T12:00:28.577203+0000","last_unstale":"2024-05-26T12:01:07.948649+0000","last_undegraded":"2024-05-26T12:01:07.948649+0000","last_fullsized":"2024-05-26T12:01:07.948649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:22:54.906432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946715+0000","last_change":"2024-05-26T12:00:32.612029+0000","last_active":"2024-05-26T12:01:07.946715+0000","last_peered":"2024-05-26T12:01:07.946715+0000","last_clean":"2024-05-26T12:01:07.946715+0000","last_became_active":"2024-05-26T12:00:32.611660+0000","last_became_peered":"2024-05-26T12:00:32.611660+0000","last_unstale":"2024-05-26T12:01:07.946715+0000","last_undegraded":"2024-05-26T12:01:07.946715+0000","last_fullsized":"2024-05-26T12:01:07.946715+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:15:27.278693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":19,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.960692+0000","last_change":"2024-05-26T12:01:07.960293+0000","last_active":"2024-05-26T12:01:07.960692+0000","last_peered":"2024-05-26T12:01:07.960692+0000","last_clean":"2024-05-26T12:01:07.960692+0000","last_became_active":"2024-05-26T12:01:07.947178+0000","last_became_peered":"2024-05-26T12:01:07.947178+0000","last_unstale":"2024-05-26T12:01:07.960692+0000","last_undegraded":"2024-05-26T12:01:07.960692+0000","last_fullsized":"2024-05-26T12:01:07.960692+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:28:23.807776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948672+0000","last_change":"2024-05-26T12:01:07.948672+0000","last_active":"2024-05-26T12:01:07.948672+0000","last_peered":"2024-05-26T12:01:07.948672+0000","last_clean":"2024-05-26T12:01:07.948672+0000","last_became_active":"2024-05-26T12:01:07.948285+0000","last_became_peered":"2024-05-26T12:01:07.948285+0000","last_unstale":"2024-05-26T12:01:07.948672+0000","last_undegraded":"2024-05-26T12:01:07.948672+0000","last_fullsized":"2024-05-26T12:01:07.948672+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:49:39.764352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:18.548422+0000","last_change":"2024-05-26T12:00:28.578507+0000","last_active":"2024-05-26T12:01:18.548422+0000","last_peered":"2024-05-26T12:01:18.548422+0000","last_clean":"2024-05-26T12:01:18.548422+0000","last_became_active":"2024-05-26T12:00:28.578131+0000","last_became_peered":"2024-05-26T12:00:28.578131+0000","last_unstale":"2024-05-26T12:01:18.548422+0000","last_undegraded":"2024-05-26T12:01:18.548422+0000","last_fullsized":"2024-05-26T12:01:18.548422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:14:52.015296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":31,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.943799+0000","last_change":"2024-05-26T12:00:32.618061+0000","last_active":"2024-05-26T12:01:07.943799+0000","last_peered":"2024-05-26T12:01:07.943799+0000","last_clean":"2024-05-26T12:01:07.943799+0000","last_became_active":"2024-05-26T12:00:32.617320+0000","last_became_peered":"2024-05-26T12:00:32.617320+0000","last_unstale":"2024-05-26T12:01:07.943799+0000","last_undegraded":"2024-05-26T12:01:07.943799+0000","last_fullsized":"2024-05-26T12:01:07.943799+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:57:33.972120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":44,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945355+0000","last_change":"2024-05-26T12:00:28.672890+0000","last_active":"2024-05-26T12:01:07.945355+0000","last_peered":"2024-05-26T12:01:07.945355+0000","last_clean":"2024-05-26T12:01:07.945355+0000","last_became_active":"2024-05-26T12:00:28.672523+0000","last_became_peered":"2024-05-26T12:00:28.672523+0000","last_unstale":"2024-05-26T12:01:07.945355+0000","last_undegraded":"2024-05-26T12:01:07.945355+0000","last_fullsized":"2024-05-26T12:01:07.945355+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:46:00.281960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945104+0000","last_change":"2024-05-26T12:00:32.618020+0000","last_active":"2024-05-26T12:01:07.945104+0000","last_peered":"2024-05-26T12:01:07.945104+0000","last_clean":"2024-05-26T12:01:07.945104+0000","last_became_active":"2024-05-26T12:00:32.617147+0000","last_became_peered":"2024-05-26T12:00:32.617147+0000","last_unstale":"2024-05-26T12:01:07.945104+0000","last_undegraded":"2024-05-26T12:01:07.945104+0000","last_fullsized":"2024-05-26T12:01:07.945104+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:54:06.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945919+0000","last_change":"2024-05-26T12:00:28.573265+0000","last_active":"2024-05-26T12:01:07.945919+0000","last_peered":"2024-05-26T12:01:07.945919+0000","last_clean":"2024-05-26T12:01:07.945919+0000","last_became_active":"2024-05-26T12:00:28.572806+0000","last_became_peered":"2024-05-26T12:00:28.572806+0000","last_unstale":"2024-05-26T12:01:07.945919+0000","last_undegraded":"2024-05-26T12:01:07.945919+0000","last_fullsized":"2024-05-26T12:01:07.945919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:20:11.192942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470842+0000","last_change":"2024-05-26T12:01:08.470842+0000","last_active":"2024-05-26T12:01:08.470842+0000","last_peered":"2024-05-26T12:01:08.470842+0000","last_clean":"2024-05-26T12:01:08.470842+0000","last_became_active":"2024-05-26T12:01:08.470548+0000","last_became_peered":"2024-05-26T12:01:08.470548+0000","last_unstale":"2024-05-26T12:01:08.470842+0000","last_undegraded":"2024-05-26T12:01:08.470842+0000","last_fullsized":"2024-05-26T12:01:08.470842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:06:11.284570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941944+0000","last_change":"2024-05-26T12:00:28.672972+0000","last_active":"2024-05-26T12:01:07.941944+0000","last_peered":"2024-05-26T12:01:07.941944+0000","last_clean":"2024-05-26T12:01:07.941944+0000","last_became_active":"2024-05-26T12:00:28.672626+0000","last_became_peered":"2024-05-26T12:00:28.672626+0000","last_unstale":"2024-05-26T12:01:07.941944+0000","last_undegraded":"2024-05-26T12:01:07.941944+0000","last_fullsized":"2024-05-26T12:01:07.941944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:34:31.449986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944509+0000","last_change":"2024-05-26T12:00:32.622325+0000","last_active":"2024-05-26T12:01:07.944509+0000","last_peered":"2024-05-26T12:01:07.944509+0000","last_clean":"2024-05-26T12:01:07.944509+0000","last_became_active":"2024-05-26T12:00:32.621970+0000","last_became_peered":"2024-05-26T12:00:32.621970+0000","last_unstale":"2024-05-26T12:01:07.944509+0000","last_undegraded":"2024-05-26T12:01:07.944509+0000","last_fullsized":"2024-05-26T12:01:07.944509+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:32:36.252057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:15.543822+0000","last_change":"2024-05-26T12:00:28.580235+0000","last_active":"2024-05-26T12:01:15.543822+0000","last_peered":"2024-05-26T12:01:15.543822+0000","last_clean":"2024-05-26T12:01:15.543822+0000","last_became_active":"2024-05-26T12:00:28.579812+0000","last_became_peered":"2024-05-26T12:00:28.579812+0000","last_unstale":"2024-05-26T12:01:15.543822+0000","last_undegraded":"2024-05-26T12:01:15.543822+0000","last_fullsized":"2024-05-26T12:01:15.543822+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:10:06.107316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":27,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944229+0000","last_change":"2024-05-26T12:00:32.625187+0000","last_active":"2024-05-26T12:01:07.944229+0000","last_peered":"2024-05-26T12:01:07.944229+0000","last_clean":"2024-05-26T12:01:07.944229+0000","last_became_active":"2024-05-26T12:00:32.624833+0000","last_became_peered":"2024-05-26T12:00:32.624833+0000","last_unstale":"2024-05-26T12:01:07.944229+0000","last_undegraded":"2024-05-26T12:01:07.944229+0000","last_fullsized":"2024-05-26T12:01:07.944229+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:49:06.502017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.950791+0000","last_change":"2024-05-26T12:01:07.950791+0000","last_active":"2024-05-26T12:01:07.950791+0000","last_peered":"2024-05-26T12:01:07.950791+0000","last_clean":"2024-05-26T12:01:07.950791+0000","last_became_active":"2024-05-26T12:01:07.950397+0000","last_became_peered":"2024-05-26T12:01:07.950397+0000","last_unstale":"2024-05-26T12:01:07.950791+0000","last_undegraded":"2024-05-26T12:01:07.950791+0000","last_fullsized":"2024-05-26T12:01:07.950791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:20:15.193328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:14.542533+0000","last_change":"2024-05-26T12:00:28.671869+0000","last_active":"2024-05-26T12:01:14.542533+0000","last_peered":"2024-05-26T12:01:14.542533+0000","last_clean":"2024-05-26T12:01:14.542533+0000","last_became_active":"2024-05-26T12:00:28.671529+0000","last_became_peered":"2024-05-26T12:00:28.671529+0000","last_unstale":"2024-05-26T12:01:14.542533+0000","last_undegraded":"2024-05-26T12:01:14.542533+0000","last_fullsized":"2024-05-26T12:01:14.542533+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:17.224354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":25,"num_read_kb":25,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940175+0000","last_change":"2024-05-26T12:00:32.621164+0000","last_active":"2024-05-26T12:01:07.940175+0000","last_peered":"2024-05-26T12:01:07.940175+0000","last_clean":"2024-05-26T12:01:07.940175+0000","last_became_active":"2024-05-26T12:00:32.618422+0000","last_became_peered":"2024-05-26T12:00:32.618422+0000","last_unstale":"2024-05-26T12:01:07.940175+0000","last_undegraded":"2024-05-26T12:01:07.940175+0000","last_fullsized":"2024-05-26T12:01:07.940175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:33:29.567862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470099+0000","last_change":"2024-05-26T12:00:28.579949+0000","last_active":"2024-05-26T12:01:08.470099+0000","last_peered":"2024-05-26T12:01:08.470099+0000","last_clean":"2024-05-26T12:01:08.470099+0000","last_became_active":"2024-05-26T12:00:28.579587+0000","last_became_peered":"2024-05-26T12:00:28.579587+0000","last_unstale":"2024-05-26T12:01:08.470099+0000","last_undegraded":"2024-05-26T12:01:08.470099+0000","last_fullsized":"2024-05-26T12:01:08.470099+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:26:18.491139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944176+0000","last_change":"2024-05-26T11:59:02.705869+0000","last_active":"2024-05-26T12:01:07.944176+0000","last_peered":"2024-05-26T12:01:07.944176+0000","last_clean":"2024-05-26T12:01:07.944176+0000","last_became_active":"2024-05-26T11:59:02.700918+0000","last_became_peered":"2024-05-26T11:59:02.700918+0000","last_unstale":"2024-05-26T12:01:07.944176+0000","last_undegraded":"2024-05-26T12:01:07.944176+0000","last_fullsized":"2024-05-26T12:01:07.944176+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_clean_scrub_stamp":"2024-05-26T11:55:35.143931+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:01:24.735040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946403+0000","last_change":"2024-05-26T12:00:32.606843+0000","last_active":"2024-05-26T12:01:07.946403+0000","last_peered":"2024-05-26T12:01:07.946403+0000","last_clean":"2024-05-26T12:01:07.946403+0000","last_became_active":"2024-05-26T12:00:32.606552+0000","last_became_peered":"2024-05-26T12:00:32.606552+0000","last_unstale":"2024-05-26T12:01:07.946403+0000","last_undegraded":"2024-05-26T12:01:07.946403+0000","last_fullsized":"2024-05-26T12:01:07.946403+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:43:03.113199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944060+0000","last_change":"2024-05-26T12:00:28.581819+0000","last_active":"2024-05-26T12:01:07.944060+0000","last_peered":"2024-05-26T12:01:07.944060+0000","last_clean":"2024-05-26T12:01:07.944060+0000","last_became_active":"2024-05-26T12:00:28.581430+0000","last_became_peered":"2024-05-26T12:00:28.581430+0000","last_unstale":"2024-05-26T12:01:07.944060+0000","last_undegraded":"2024-05-26T12:01:07.944060+0000","last_fullsized":"2024-05-26T12:01:07.944060+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:13:42.556238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946242+0000","last_change":"2024-05-26T12:00:28.672396+0000","last_active":"2024-05-26T12:01:07.946242+0000","last_peered":"2024-05-26T12:01:07.946242+0000","last_clean":"2024-05-26T12:01:07.946242+0000","last_became_active":"2024-05-26T12:00:28.672045+0000","last_became_peered":"2024-05-26T12:00:28.672045+0000","last_unstale":"2024-05-26T12:01:07.946242+0000","last_undegraded":"2024-05-26T12:01:07.946242+0000","last_fullsized":"2024-05-26T12:01:07.946242+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:02:25.925210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946831+0000","last_change":"2024-05-26T12:00:32.606876+0000","last_active":"2024-05-26T12:01:07.946831+0000","last_peered":"2024-05-26T12:01:07.946831+0000","last_clean":"2024-05-26T12:01:07.946831+0000","last_became_active":"2024-05-26T12:00:32.606161+0000","last_became_peered":"2024-05-26T12:00:32.606161+0000","last_unstale":"2024-05-26T12:01:07.946831+0000","last_undegraded":"2024-05-26T12:01:07.946831+0000","last_fullsized":"2024-05-26T12:01:07.946831+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:47:02.451447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'15","reported_seq":57,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:14.625301+0000","last_change":"2024-05-26T12:01:07.955816+0000","last_active":"2024-05-26T12:01:14.625301+0000","last_peered":"2024-05-26T12:01:14.625301+0000","last_clean":"2024-05-26T12:01:14.625301+0000","last_became_active":"2024-05-26T12:01:07.947443+0000","last_became_peered":"2024-05-26T12:01:07.947443+0000","last_unstale":"2024-05-26T12:01:14.625301+0000","last_undegraded":"2024-05-26T12:01:14.625301+0000","last_fullsized":"2024-05-26T12:01:14.625301+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:30:00.005339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940312+0000","last_change":"2024-05-26T12:00:32.616713+0000","last_active":"2024-05-26T12:01:07.940312+0000","last_peered":"2024-05-26T12:01:07.940312+0000","last_clean":"2024-05-26T12:01:07.940312+0000","last_became_active":"2024-05-26T12:00:32.616382+0000","last_became_peered":"2024-05-26T12:00:32.616382+0000","last_unstale":"2024-05-26T12:01:07.940312+0000","last_undegraded":"2024-05-26T12:01:07.940312+0000","last_fullsized":"2024-05-26T12:01:07.940312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:12:42.795794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":34,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:08.470352+0000","last_change":"2024-05-26T12:00:28.971347+0000","last_active":"2024-05-26T12:01:08.470352+0000","last_peered":"2024-05-26T12:01:08.470352+0000","last_clean":"2024-05-26T12:01:08.470352+0000","last_became_active":"2024-05-26T12:00:28.970911+0000","last_became_peered":"2024-05-26T12:00:28.970911+0000","last_unstale":"2024-05-26T12:01:08.470352+0000","last_undegraded":"2024-05-26T12:01:08.470352+0000","last_fullsized":"2024-05-26T12:01:08.470352+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:47:45.306875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940960+0000","last_change":"2024-05-26T12:00:32.614238+0000","last_active":"2024-05-26T12:01:07.940960+0000","last_peered":"2024-05-26T12:01:07.940960+0000","last_clean":"2024-05-26T12:01:07.940960+0000","last_became_active":"2024-05-26T12:00:32.607018+0000","last_became_peered":"2024-05-26T12:00:32.607018+0000","last_unstale":"2024-05-26T12:01:07.940960+0000","last_undegraded":"2024-05-26T12:01:07.940960+0000","last_fullsized":"2024-05-26T12:01:07.940960+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:31:06.250615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.947096+0000","last_change":"2024-05-26T12:00:28.671961+0000","last_active":"2024-05-26T12:01:07.947096+0000","last_peered":"2024-05-26T12:01:07.947096+0000","last_clean":"2024-05-26T12:01:07.947096+0000","last_became_active":"2024-05-26T12:00:28.671672+0000","last_became_peered":"2024-05-26T12:00:28.671672+0000","last_unstale":"2024-05-26T12:01:07.947096+0000","last_undegraded":"2024-05-26T12:01:07.947096+0000","last_fullsized":"2024-05-26T12:01:07.947096+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:05.646044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945323+0000","last_change":"2024-05-26T12:00:32.611432+0000","last_active":"2024-05-26T12:01:07.945323+0000","last_peered":"2024-05-26T12:01:07.945323+0000","last_clean":"2024-05-26T12:01:07.945323+0000","last_became_active":"2024-05-26T12:00:32.610686+0000","last_became_peered":"2024-05-26T12:00:32.610686+0000","last_unstale":"2024-05-26T12:01:07.945323+0000","last_undegraded":"2024-05-26T12:01:07.945323+0000","last_fullsized":"2024-05-26T12:01:07.945323+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:52:36.459552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":33,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939974+0000","last_change":"2024-05-26T12:00:28.576723+0000","last_active":"2024-05-26T12:01:07.939974+0000","last_peered":"2024-05-26T12:01:07.939974+0000","last_clean":"2024-05-26T12:01:07.939974+0000","last_became_active":"2024-05-26T12:00:28.576380+0000","last_became_peered":"2024-05-26T12:00:28.576380+0000","last_unstale":"2024-05-26T12:01:07.939974+0000","last_undegraded":"2024-05-26T12:01:07.939974+0000","last_fullsized":"2024-05-26T12:01:07.939974+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:49:25.611830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942224+0000","last_change":"2024-05-26T12:00:32.621124+0000","last_active":"2024-05-26T12:01:07.942224+0000","last_peered":"2024-05-26T12:01:07.942224+0000","last_clean":"2024-05-26T12:01:07.942224+0000","last_became_active":"2024-05-26T12:00:32.620740+0000","last_became_peered":"2024-05-26T12:00:32.620740+0000","last_unstale":"2024-05-26T12:01:07.942224+0000","last_undegraded":"2024-05-26T12:01:07.942224+0000","last_fullsized":"2024-05-26T12:01:07.942224+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:15:16.123821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948705+0000","last_change":"2024-05-26T12:00:28.572222+0000","last_active":"2024-05-26T12:01:07.948705+0000","last_peered":"2024-05-26T12:01:07.948705+0000","last_clean":"2024-05-26T12:01:07.948705+0000","last_became_active":"2024-05-26T12:00:28.571459+0000","last_became_peered":"2024-05-26T12:00:28.571459+0000","last_unstale":"2024-05-26T12:01:07.948705+0000","last_undegraded":"2024-05-26T12:01:07.948705+0000","last_fullsized":"2024-05-26T12:01:07.948705+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:49:56.234550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940179+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:01:07.940179+0000","last_peered":"2024-05-26T12:01:07.940179+0000","last_clean":"2024-05-26T12:01:07.940179+0000","last_became_active":"2024-05-26T12:00:32.617579+0000","last_became_peered":"2024-05-26T12:00:32.617579+0000","last_unstale":"2024-05-26T12:01:07.940179+0000","last_undegraded":"2024-05-26T12:01:07.940179+0000","last_fullsized":"2024-05-26T12:01:07.940179+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:46:57.045155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945531+0000","last_change":"2024-05-26T12:00:28.584555+0000","last_active":"2024-05-26T12:01:07.945531+0000","last_peered":"2024-05-26T12:01:07.945531+0000","last_clean":"2024-05-26T12:01:07.945531+0000","last_became_active":"2024-05-26T12:00:28.583750+0000","last_became_peered":"2024-05-26T12:00:28.583750+0000","last_unstale":"2024-05-26T12:01:07.945531+0000","last_undegraded":"2024-05-26T12:01:07.945531+0000","last_fullsized":"2024-05-26T12:01:07.945531+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:26:50.312369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944253+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:01:07.944253+0000","last_peered":"2024-05-26T12:01:07.944253+0000","last_clean":"2024-05-26T12:01:07.944253+0000","last_became_active":"2024-05-26T12:00:32.615758+0000","last_became_peered":"2024-05-26T12:00:32.615758+0000","last_unstale":"2024-05-26T12:01:07.944253+0000","last_undegraded":"2024-05-26T12:01:07.944253+0000","last_fullsized":"2024-05-26T12:01:07.944253+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:17:52.776655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:17.547434+0000","last_change":"2024-05-26T12:00:28.971186+0000","last_active":"2024-05-26T12:01:17.547434+0000","last_peered":"2024-05-26T12:01:17.547434+0000","last_clean":"2024-05-26T12:01:17.547434+0000","last_became_active":"2024-05-26T12:00:28.970706+0000","last_became_peered":"2024-05-26T12:00:28.970706+0000","last_unstale":"2024-05-26T12:01:17.547434+0000","last_undegraded":"2024-05-26T12:01:17.547434+0000","last_fullsized":"2024-05-26T12:01:17.547434+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:39:12.139262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":29,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945309+0000","last_change":"2024-05-26T12:00:32.623244+0000","last_active":"2024-05-26T12:01:07.945309+0000","last_peered":"2024-05-26T12:01:07.945309+0000","last_clean":"2024-05-26T12:01:07.945309+0000","last_became_active":"2024-05-26T12:00:32.622057+0000","last_became_peered":"2024-05-26T12:00:32.622057+0000","last_unstale":"2024-05-26T12:01:07.945309+0000","last_undegraded":"2024-05-26T12:01:07.945309+0000","last_fullsized":"2024-05-26T12:01:07.945309+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:37:09.871626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.945442+0000","last_change":"2024-05-26T12:00:28.576115+0000","last_active":"2024-05-26T12:01:07.945442+0000","last_peered":"2024-05-26T12:01:07.945442+0000","last_clean":"2024-05-26T12:01:07.945442+0000","last_became_active":"2024-05-26T12:00:28.575704+0000","last_became_peered":"2024-05-26T12:00:28.575704+0000","last_unstale":"2024-05-26T12:01:07.945442+0000","last_undegraded":"2024-05-26T12:01:07.945442+0000","last_fullsized":"2024-05-26T12:01:07.945442+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:28:05.142390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":21,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944558+0000","last_change":"2024-05-26T12:00:32.611997+0000","last_active":"2024-05-26T12:01:07.944558+0000","last_peered":"2024-05-26T12:01:07.944558+0000","last_clean":"2024-05-26T12:01:07.944558+0000","last_became_active":"2024-05-26T12:00:32.611646+0000","last_became_peered":"2024-05-26T12:00:32.611646+0000","last_unstale":"2024-05-26T12:01:07.944558+0000","last_undegraded":"2024-05-26T12:01:07.944558+0000","last_fullsized":"2024-05-26T12:01:07.944558+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:50:15.856048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946931+0000","last_change":"2024-05-26T12:00:28.576500+0000","last_active":"2024-05-26T12:01:07.946931+0000","last_peered":"2024-05-26T12:01:07.946931+0000","last_clean":"2024-05-26T12:01:07.946931+0000","last_became_active":"2024-05-26T12:00:28.576171+0000","last_became_peered":"2024-05-26T12:00:28.576171+0000","last_unstale":"2024-05-26T12:01:07.946931+0000","last_undegraded":"2024-05-26T12:01:07.946931+0000","last_fullsized":"2024-05-26T12:01:07.946931+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:44:54.786997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944854+0000","last_change":"2024-05-26T12:00:32.617550+0000","last_active":"2024-05-26T12:01:07.944854+0000","last_peered":"2024-05-26T12:01:07.944854+0000","last_clean":"2024-05-26T12:01:07.944854+0000","last_became_active":"2024-05-26T12:00:32.616821+0000","last_became_peered":"2024-05-26T12:00:32.616821+0000","last_unstale":"2024-05-26T12:01:07.944854+0000","last_undegraded":"2024-05-26T12:01:07.944854+0000","last_fullsized":"2024-05-26T12:01:07.944854+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:49:59.956694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942311+0000","last_change":"2024-05-26T12:00:32.623314+0000","last_active":"2024-05-26T12:01:07.942311+0000","last_peered":"2024-05-26T12:01:07.942311+0000","last_clean":"2024-05-26T12:01:07.942311+0000","last_became_active":"2024-05-26T12:00:32.622187+0000","last_became_peered":"2024-05-26T12:00:32.622187+0000","last_unstale":"2024-05-26T12:01:07.942311+0000","last_undegraded":"2024-05-26T12:01:07.942311+0000","last_fullsized":"2024-05-26T12:01:07.942311+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:19:08.026803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948805+0000","last_change":"2024-05-26T12:00:32.614814+0000","last_active":"2024-05-26T12:01:07.948805+0000","last_peered":"2024-05-26T12:01:07.948805+0000","last_clean":"2024-05-26T12:01:07.948805+0000","last_became_active":"2024-05-26T12:00:32.614439+0000","last_became_peered":"2024-05-26T12:00:32.614439+0000","last_unstale":"2024-05-26T12:01:07.948805+0000","last_undegraded":"2024-05-26T12:01:07.948805+0000","last_fullsized":"2024-05-26T12:01:07.948805+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:56:10.817914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948574+0000","last_change":"2024-05-26T12:00:32.610170+0000","last_active":"2024-05-26T12:01:07.948574+0000","last_peered":"2024-05-26T12:01:07.948574+0000","last_clean":"2024-05-26T12:01:07.948574+0000","last_became_active":"2024-05-26T12:00:32.609783+0000","last_became_peered":"2024-05-26T12:00:32.609783+0000","last_unstale":"2024-05-26T12:01:07.948574+0000","last_undegraded":"2024-05-26T12:01:07.948574+0000","last_fullsized":"2024-05-26T12:01:07.948574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:33:52.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.941107+0000","last_change":"2024-05-26T12:00:32.609819+0000","last_active":"2024-05-26T12:01:07.941107+0000","last_peered":"2024-05-26T12:01:07.941107+0000","last_clean":"2024-05-26T12:01:07.941107+0000","last_became_active":"2024-05-26T12:00:32.609016+0000","last_became_peered":"2024-05-26T12:00:32.609016+0000","last_unstale":"2024-05-26T12:01:07.941107+0000","last_undegraded":"2024-05-26T12:01:07.941107+0000","last_fullsized":"2024-05-26T12:01:07.941107+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:10:09.582603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946791+0000","last_change":"2024-05-26T12:00:32.608904+0000","last_active":"2024-05-26T12:01:07.946791+0000","last_peered":"2024-05-26T12:01:07.946791+0000","last_clean":"2024-05-26T12:01:07.946791+0000","last_became_active":"2024-05-26T12:00:32.608505+0000","last_became_peered":"2024-05-26T12:00:32.608505+0000","last_unstale":"2024-05-26T12:01:07.946791+0000","last_undegraded":"2024-05-26T12:01:07.946791+0000","last_fullsized":"2024-05-26T12:01:07.946791+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:34:09.931046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942574+0000","last_change":"2024-05-26T12:00:32.618370+0000","last_active":"2024-05-26T12:01:07.942574+0000","last_peered":"2024-05-26T12:01:07.942574+0000","last_clean":"2024-05-26T12:01:07.942574+0000","last_became_active":"2024-05-26T12:00:32.618022+0000","last_became_peered":"2024-05-26T12:00:32.618022+0000","last_unstale":"2024-05-26T12:01:07.942574+0000","last_undegraded":"2024-05-26T12:01:07.942574+0000","last_fullsized":"2024-05-26T12:01:07.942574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:45:12.652460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.949432+0000","last_change":"2024-05-26T12:00:32.612319+0000","last_active":"2024-05-26T12:01:07.949432+0000","last_peered":"2024-05-26T12:01:07.949432+0000","last_clean":"2024-05-26T12:01:07.949432+0000","last_became_active":"2024-05-26T12:00:32.611973+0000","last_became_peered":"2024-05-26T12:00:32.611973+0000","last_unstale":"2024-05-26T12:01:07.949432+0000","last_undegraded":"2024-05-26T12:01:07.949432+0000","last_fullsized":"2024-05-26T12:01:07.949432+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:23:48.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.940298+0000","last_change":"2024-05-26T12:00:32.613461+0000","last_active":"2024-05-26T12:01:07.940298+0000","last_peered":"2024-05-26T12:01:07.940298+0000","last_clean":"2024-05-26T12:01:07.940298+0000","last_became_active":"2024-05-26T12:00:32.613068+0000","last_became_peered":"2024-05-26T12:00:32.613068+0000","last_unstale":"2024-05-26T12:01:07.940298+0000","last_undegraded":"2024-05-26T12:01:07.940298+0000","last_fullsized":"2024-05-26T12:01:07.940298+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:17:59.606664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.944957+0000","last_change":"2024-05-26T12:00:32.615583+0000","last_active":"2024-05-26T12:01:07.944957+0000","last_peered":"2024-05-26T12:01:07.944957+0000","last_clean":"2024-05-26T12:01:07.944957+0000","last_became_active":"2024-05-26T12:00:32.611502+0000","last_became_peered":"2024-05-26T12:00:32.611502+0000","last_unstale":"2024-05-26T12:01:07.944957+0000","last_undegraded":"2024-05-26T12:01:07.944957+0000","last_fullsized":"2024-05-26T12:01:07.944957+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:26:47.772010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.939923+0000","last_change":"2024-05-26T12:00:32.612190+0000","last_active":"2024-05-26T12:01:07.939923+0000","last_peered":"2024-05-26T12:01:07.939923+0000","last_clean":"2024-05-26T12:01:07.939923+0000","last_became_active":"2024-05-26T12:00:32.611843+0000","last_became_peered":"2024-05-26T12:00:32.611843+0000","last_unstale":"2024-05-26T12:01:07.939923+0000","last_undegraded":"2024-05-26T12:01:07.939923+0000","last_fullsized":"2024-05-26T12:01:07.939923+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:23:05.244741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.948397+0000","last_change":"2024-05-26T12:00:32.620758+0000","last_active":"2024-05-26T12:01:07.948397+0000","last_peered":"2024-05-26T12:01:07.948397+0000","last_clean":"2024-05-26T12:01:07.948397+0000","last_became_active":"2024-05-26T12:00:32.620401+0000","last_became_peered":"2024-05-26T12:00:32.620401+0000","last_unstale":"2024-05-26T12:01:07.948397+0000","last_undegraded":"2024-05-26T12:01:07.948397+0000","last_fullsized":"2024-05-26T12:01:07.948397+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:59:18.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.946941+0000","last_change":"2024-05-26T12:00:32.615615+0000","last_active":"2024-05-26T12:01:07.946941+0000","last_peered":"2024-05-26T12:01:07.946941+0000","last_clean":"2024-05-26T12:01:07.946941+0000","last_became_active":"2024-05-26T12:00:32.615304+0000","last_became_peered":"2024-05-26T12:00:32.615304+0000","last_unstale":"2024-05-26T12:01:07.946941+0000","last_undegraded":"2024-05-26T12:01:07.946941+0000","last_fullsized":"2024-05-26T12:01:07.946941+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:05:22.568910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-05-26T12:01:07.942360+0000","last_change":"2024-05-26T12:00:32.627209+0000","last_active":"2024-05-26T12:01:07.942360+0000","last_peered":"2024-05-26T12:01:07.942360+0000","last_clean":"2024-05-26T12:01:07.942360+0000","last_became_active":"2024-05-26T12:00:32.626851+0000","last_became_peered":"2024-05-26T12:00:32.626851+0000","last_unstale":"2024-05-26T12:01:07.942360+0000","last_undegraded":"2024-05-26T12:01:07.942360+0000","last_fullsized":"2024-05-26T12:01:07.942360+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:05:40.754565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213552,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66575,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":303,"num_read_kb":219,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":28173,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":199725,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678041,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32040,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713112,"statfs":{"total":95995035648,"available":95962226688,"internally_reserved":0,"allocated":884736,"data_stored":429000,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874270,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1079701,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037796,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":428840,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":429218,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":954368,"data_stored":1021736,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593779,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27888,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717264,"statfs":{"total":95995035648,"available":95966478336,"internally_reserved":0,"allocated":888832,"data_stored":428948,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574925,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":489623,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738448,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36232,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708920,"statfs":{"total":95995035648,"available":95957934080,"internally_reserved":0,"allocated":909312,"data_stored":429896,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27896,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717256,"statfs":{"total":95995035648,"available":95966470144,"internally_reserved":0,"allocated":901120,"data_stored":429690,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149763,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27900,"kb_used_data":884,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717252,"statfs":{"total":95995035648,"available":95966466048,"internally_reserved":0,"allocated":905216,"data_stored":428382,"data_compressed":5779,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953534,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":483923,"data_compressed":10339,"data_compressed_allocated":16384,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790008,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":962560,"data_stored":1020933,"data_compressed":41911,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60462,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T12:01:23.502 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-26T12:01:23.503 INFO:tasks.cephadm:Setting up client nodes... 2024-05-26T12:01:23.503 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-26T12:01:23.750 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:24.130 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:23 smithi156 ceph-mon[32025]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:24.130 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:23 smithi156 ceph-mon[32025]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:24.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:23 smithi192 ceph-mon[36545]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:24.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:23 smithi192 ceph-mon[36545]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:24.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:23 smithi186 ceph-mon[36412]: from='client.14739 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:01:24.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:23 smithi186 ceph-mon[36412]: pgmap v319: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.2 KiB/s wr, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 -- 172.21.15.156:0/3138385236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 msgr2=0x7fbb4c103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 --2- 172.21.15.156:0/3138385236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c103b20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c0099c0 tx=0x7fbb3c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 -- 172.21.15.156:0/3138385236 shutdown_connections 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 --2- 172.21.15.156:0/3138385236 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c104060 0x7fbb4c10a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 --2- 172.21.15.156:0/3138385236 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c1024c0 0x7fbb4c1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 --2- 172.21.15.156:0/3138385236 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c103b20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.486+0000 7fbb54224700 1 -- 172.21.15.156:0/3138385236 >> 172.21.15.156:0/3138385236 conn(0x7fbb4c0fdaa0 msgr2=0x7fbb4c0ffec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- 172.21.15.156:0/3138385236 shutdown_connections 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- 172.21.15.156:0/3138385236 wait complete. 2024-05-26T12:01:25.489 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 Processor -- start 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- start start 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c1024c0 0x7fbb4c199690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c104060 0x7fbb4c1a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c10df60 con 0x7fbb4c1036c0 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c10dde0 con 0x7fbb4c1024c0 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.487+0000 7fbb54224700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fbb4c10e0e0 con 0x7fbb4c104060 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42892/0 (socket says 172.21.15.156:42892) 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 -- 172.21.15.156:0/3846543407 learned_addr learned my addr 172.21.15.156:0/3846543407 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb51fc0700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c1024c0 0x7fbb4c199690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:25.490 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb527c1700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c104060 0x7fbb4c1a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 -- 172.21.15.156:0/3846543407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c104060 msgr2=0x7fbb4c1a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c104060 0x7fbb4c1a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 -- 172.21.15.156:0/3846543407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c1024c0 msgr2=0x7fbb4c199690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c1024c0 0x7fbb4c199690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 -- 172.21.15.156:0/3846543407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb4c1a12f0 con 0x7fbb4c1036c0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb517bf700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c005910 tx=0x7fbb3c005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c027070 con 0x7fbb4c1036c0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbb3c0041a0 con 0x7fbb4c1036c0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c022dd0 con 0x7fbb4c1036c0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb4c1a1580 con 0x7fbb4c1036c0 2024-05-26T12:01:25.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.488+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb4c1a1a60 con 0x7fbb4c1036c0 2024-05-26T12:01:25.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.489+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb4c051a80 con 0x7fbb4c1036c0 2024-05-26T12:01:25.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.491+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fbb3c02a070 con 0x7fbb4c1036c0 2024-05-26T12:01:25.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.491+0000 7fbb42ffd700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 0x7fbb380b4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:25.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.492+0000 7fbb51fc0700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 0x7fbb380b4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:25.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.492+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fbb3c0e33e0 con 0x7fbb4c1036c0 2024-05-26T12:01:25.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.492+0000 7fbb51fc0700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 0x7fbb380b4e70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005950 tx=0x7fbb4800a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:25.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.495+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fbb3c013070 con 0x7fbb4c1036c0 2024-05-26T12:01:25.785 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.785+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fbb4c1a1fe0 con 0x7fbb4c1036c0 2024-05-26T12:01:25.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.787+0000 7fbb42ffd700 1 -- 172.21.15.156:0/3846543407 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fbb3c0ab020 con 0x7fbb4c1036c0 2024-05-26T12:01:25.787 INFO:teuthology.orchestra.run.smithi156.stdout:[client.0] 2024-05-26T12:01:25.787 INFO:teuthology.orchestra.run.smithi156.stdout: key = AQCVJFNmCiHhLhAAFSrP5SsRmsYQvhO4zmjXNA== 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 msgr2=0x7fbb380b4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 0x7fbb380b4e70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fbb48005950 tx=0x7fbb4800a300 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 msgr2=0x7fbb4c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c005910 tx=0x7fbb3c005ef0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 shutdown_connections 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fbb380b29b0 0x7fbb380b4e70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fbb4c104060 0x7fbb4c1a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4c1024c0 0x7fbb4c199690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 --2- 172.21.15.156:0/3846543407 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fbb4c1036c0 0x7fbb4c199bd0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 >> 172.21.15.156:0/3846543407 conn(0x7fbb4c0fdaa0 msgr2=0x7fbb4c0ff4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 shutdown_connections 2024-05-26T12:01:25.790 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:25.789+0000 7fbb54224700 1 -- 172.21.15.156:0/3846543407 wait complete. 2024-05-26T12:01:26.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:25 smithi156 ceph-mon[32025]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:26.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:25 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-26T12:01:26.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:25 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-26T12:01:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:25 smithi192 ceph-mon[36545]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:25 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-26T12:01:26.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:25 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-26T12:01:26.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:25 smithi186 ceph-mon[36412]: pgmap v320: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-05-26T12:01:26.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:25 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-26T12:01:26.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:25 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3846543407' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-26T12:01:26.498 DEBUG:teuthology.orchestra.run.smithi156:> set -ex 2024-05-26T12:01:26.498 DEBUG:teuthology.orchestra.run.smithi156:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-26T12:01:26.498 DEBUG:teuthology.orchestra.run.smithi156:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-26T12:01:26.537 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-26T12:01:26.537 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-26T12:01:26.537 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph mgr dump --format=json 2024-05-26T12:01:26.821 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:28.264 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:27 smithi156 ceph-mon[32025]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-05-26T12:01:28.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:27 smithi192 ceph-mon[36545]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-05-26T12:01:28.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:27 smithi186 ceph-mon[36412]: pgmap v321: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1023 B/s wr, 11 op/s 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 -- 172.21.15.156:0/1516655902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a941035f0 msgr2=0x7f2a94103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 --2- 172.21.15.156:0/1516655902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a941035f0 0x7f2a94103a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2a880098f0 tx=0x7f2a88009c00 comp rx=0 tx=0).stop 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 -- 172.21.15.156:0/1516655902 shutdown_connections 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 --2- 172.21.15.156:0/1516655902 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94103f90 0x7f2a94110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 --2- 172.21.15.156:0/1516655902 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a94108500 0x7f2a941088e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.483 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 --2- 172.21.15.156:0/1516655902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a941035f0 0x7f2a94103a50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 -- 172.21.15.156:0/1516655902 >> 172.21.15.156:0/1516655902 conn(0x7f2a9407a6a0 msgr2=0x7f2a9407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 -- 172.21.15.156:0/1516655902 shutdown_connections 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.480+0000 7f2a9985f700 1 -- 172.21.15.156:0/1516655902 wait complete. 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 Processor -- start 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 -- start start 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94108500 0x7f2a94199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f2a941137a0 con 0x7f2a941035f0 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.481+0000 7f2a9985f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94113620 con 0x7f2a94103f90 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9985f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f2a94113920 con 0x7f2a94108500 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:28.484 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.156:50118/0 (socket says 172.21.15.156:50118) 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a92ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42922/0 (socket says 172.21.15.156:42922) 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 -- 172.21.15.156:0/430509235 learned_addr learned my addr 172.21.15.156:0/430509235 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a937fe700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94108500 0x7f2a94199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 -- 172.21.15.156:0/430509235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94108500 msgr2=0x7f2a94199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94108500 0x7f2a94199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 -- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 msgr2=0x7f2a9419f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 -- 172.21.15.156:0/430509235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9419a1c0 con 0x7f2a94103f90 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a92ffd700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a927fc700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800f990 tx=0x7f2a8800f9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88027070 con 0x7f2a94103f90 2024-05-26T12:01:28.485 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9419a450 con 0x7f2a94103f90 2024-05-26T12:01:28.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2a88005570 con 0x7f2a94103f90 2024-05-26T12:01:28.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a88004930 con 0x7f2a94103f90 2024-05-26T12:01:28.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.482+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a941a6770 con 0x7f2a94103f90 2024-05-26T12:01:28.486 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.484+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9406a230 con 0x7f2a94103f90 2024-05-26T12:01:28.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.485+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f2a8802a070 con 0x7f2a94103f90 2024-05-26T12:01:28.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.486+0000 7f2a9885d700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 0x7f2a7c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:28.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.486+0000 7f2a92ffd700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 0x7f2a7c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:28.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.486+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f2a880e2f10 con 0x7f2a94103f90 2024-05-26T12:01:28.487 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.486+0000 7f2a92ffd700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 0x7f2a7c0b4dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a9419aba0 tx=0x7f2a84009310 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:28.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.493+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f2a88013030 con 0x7f2a94103f90 2024-05-26T12:01:28.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.787+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f2a9402e560 con 0x7f2a94103f90 2024-05-26T12:01:28.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.793+0000 7f2a9885d700 1 -- 172.21.15.156:0/430509235 <== mon.2 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+288654 (secure 0 0 0) 0x7f2a880aaaf0 con 0x7f2a94103f90 2024-05-26T12:01:28.797 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 msgr2=0x7f2a7c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 0x7f2a7c0b4dd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f2a9419aba0 tx=0x7f2a84009310 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 msgr2=0x7f2a9419fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800f990 tx=0x7f2a8800f9c0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 shutdown_connections 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f2a7c0b2910 0x7f2a7c0b4dd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f2a94108500 0x7f2a94199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f2a941035f0 0x7f2a9419f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 --2- 172.21.15.156:0/430509235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a94103f90 0x7f2a9419fde0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 >> 172.21.15.156:0/430509235 conn(0x7f2a9407a6a0 msgr2=0x7f2a94101740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 shutdown_connections 2024-05-26T12:01:28.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:28.796+0000 7f2a9985f700 1 -- 172.21.15.156:0/430509235 wait complete. 2024-05-26T12:01:29.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:28 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.045 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:28 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/430509235' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.226 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:28 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.227 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:28 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/430509235' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:28 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:28 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/430509235' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-26T12:01:29.695 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6800","nonce":3256012970},{"type":"v1","addr":"172.21.15.156:6801","nonce":3256012970}]},"active_addr":"172.21.15.156:6801/3256012970","active_change":"2024-05-26T11:53:06.708308+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.156:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":2022472305}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":3899086288}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":71937279}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":1371336780}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.156:0","nonce":3528884460}]}]} 2024-05-26T12:01:29.701 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-26T12:01:29.701 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-26T12:01:29.702 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:01:30.172 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:29 smithi156 ceph-mon[32025]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:01:30.172 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.172 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.172 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.172 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:29 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.195 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:29 smithi192 ceph-mon[36545]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:01:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:29 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:29 smithi186 ceph-mon[36412]: pgmap v322: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:01:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:30.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:29 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3126004227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 msgr2=0x7fb99c1103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:31.919 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3126004227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c1103d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fb99400b0d0 tx=0x7fb99400b3e0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3126004227 shutdown_connections 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3126004227 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c1103d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3126004227 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c109520 0x7fb99c109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3126004227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c101b30 0x7fb99c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3126004227 >> 172.21.15.156:0/3126004227 conn(0x7fb99c078ab0 msgr2=0x7fb99c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3126004227 shutdown_connections 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.917+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3126004227 wait complete. 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 Processor -- start 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 -- start start 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c101b30 0x7fb99c19f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:31.920 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c109520 0x7fb99c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c113720 con 0x7fb99c1024f0 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1135a0 con 0x7fb99c109520 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb9a1da2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb99c1138a0 con 0x7fb99c101b30 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb99affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb99affd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42942/0 (socket says 172.21.15.156:42942) 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb99affd700 1 -- 172.21.15.156:0/3943066739 learned_addr learned my addr 172.21.15.156:0/3943066739 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb99b7fe700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c101b30 0x7fb99c19f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.918+0000 7fb99bfff700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c109520 0x7fb99c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 -- 172.21.15.156:0/3943066739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c101b30 msgr2=0x7fb99c19f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c101b30 0x7fb99c19f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 -- 172.21.15.156:0/3943066739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c109520 msgr2=0x7fb99c1999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c109520 0x7fb99c1999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:31.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 -- 172.21.15.156:0/3943066739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb99c19a140 con 0x7fb99c1024f0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb99affd700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb99000d950 tx=0x7fb99000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb990016070 con 0x7fb99c1024f0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb99c19a430 con 0x7fb99c1024f0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb99000fd40 con 0x7fb99c1024f0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99000e3b0 con 0x7fb99c1024f0 2024-05-26T12:01:31.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.919+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb99c1a66f0 con 0x7fb99c1024f0 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.920+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb99c104e50 con 0x7fb99c1024f0 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.921+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fb990019070 con 0x7fb99c1024f0 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.922+0000 7fb998ff9700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 0x7fb9840b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.922+0000 7fb99b7fe700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 0x7fb9840b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.922+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fb9900e1e00 con 0x7fb99c1024f0 2024-05-26T12:01:31.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.922+0000 7fb99b7fe700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 0x7fb9840b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb99c19ab20 tx=0x7fb98c00b3c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:31.926 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:31.925+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb9900a99b0 con 0x7fb99c1024f0 2024-05-26T12:01:32.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.180+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb99c02e560 con 0x7fb99c1024f0 2024-05-26T12:01:32.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.181+0000 7fb998ff9700 1 -- 172.21.15.156:0/3943066739 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22707 (secure 0 0 0) 0x7fb9900a92b0 con 0x7fb99c1024f0 2024-05-26T12:01:32.182 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:32.182 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":81,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:01:07.937226+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:01:32.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 msgr2=0x7fb9840b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:32.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 0x7fb9840b4dd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fb99c19ab20 tx=0x7fb98c00b3c0 comp rx=0 tx=0).stop 2024-05-26T12:01:32.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 msgr2=0x7fb99c19fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:32.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb99000d950 tx=0x7fb99000dc60 comp rx=0 tx=0).stop 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 shutdown_connections 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb9840b2910 0x7fb9840b4dd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb99c109520 0x7fb99c1999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb99c101b30 0x7fb99c19f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 --2- 172.21.15.156:0/3943066739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb99c1024f0 0x7fb99c19fd60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 >> 172.21.15.156:0/3943066739 conn(0x7fb99c078ab0 msgr2=0x7fb99c100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 shutdown_connections 2024-05-26T12:01:32.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:32.184+0000 7fb9a1da2700 1 -- 172.21.15.156:0/3943066739 wait complete. 2024-05-26T12:01:32.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:31 smithi156 ceph-mon[32025]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:01:32.322 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:31 smithi192 ceph-mon[36545]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:01:32.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:31 smithi186 ceph-mon[36412]: pgmap v323: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:01:32.907 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-26T12:01:32.907 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3943066739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:01:33.147 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.158 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:33.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3943066739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:01:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:01:33.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3943066739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:01:33.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:01:34.272 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:34 smithi156 ceph-mon[32025]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:34.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:34 smithi186 ceph-mon[36412]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:34.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:34 smithi192 ceph-mon[36545]: pgmap v324: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 -- 172.21.15.156:0/3949515773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a701095b0 msgr2=0x7f6a70109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 --2- 172.21.15.156:0/3949515773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a701095b0 0x7f6a70109990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009990 tx=0x7f6a60009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 -- 172.21.15.156:0/3949515773 shutdown_connections 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 --2- 172.21.15.156:0/3949515773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a7006bbe0 0x7f6a701103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 --2- 172.21.15.156:0/3949515773 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a701095b0 0x7f6a70109990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 --2- 172.21.15.156:0/3949515773 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006b220 0x7f6a7006b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 -- 172.21.15.156:0/3949515773 >> 172.21.15.156:0/3949515773 conn(0x7f6a700793f0 msgr2=0x7f6a70079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 -- 172.21.15.156:0/3949515773 shutdown_connections 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.795+0000 7f6a7599e700 1 -- 172.21.15.156:0/3949515773 wait complete. 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 Processor -- start 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 -- start start 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006bbe0 0x7f6a7019fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a701095b0 0x7f6a70199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6a701136c0 con 0x7f6a7006b220 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6a70113540 con 0x7f6a701095b0 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a7599e700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6a70113840 con 0x7f6a7006bbe0 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:46610/0 (socket says 172.21.15.156:46610) 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 -- 172.21.15.156:0/1366998827 learned_addr learned my addr 172.21.15.156:0/1366998827 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6f7fe700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a701095b0 0x7f6a70199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:34.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6e7fc700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006bbe0 0x7f6a7019fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 -- 172.21.15.156:0/1366998827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006bbe0 msgr2=0x7f6a7019fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006bbe0 0x7f6a7019fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 -- 172.21.15.156:0/1366998827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a701095b0 msgr2=0x7f6a70199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a701095b0 0x7f6a70199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.796+0000 7f6a6effd700 1 -- 172.21.15.156:0/1366998827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7019a1a0 con 0x7f6a7006b220 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a6effd700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a600047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60027070 con 0x7f6a7006b220 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6a60003fa0 con 0x7f6a7006b220 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7019a430 con 0x7f6a7006b220 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60022db0 con 0x7f6a7006b220 2024-05-26T12:01:34.800 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.797+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a701a6700 con 0x7f6a7006b220 2024-05-26T12:01:34.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.798+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a7010ba90 con 0x7f6a7006b220 2024-05-26T12:01:34.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.800+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f6a6002a070 con 0x7f6a7006b220 2024-05-26T12:01:34.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.801+0000 7f6a7499c700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 0x7f6a580b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:34.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.801+0000 7f6a6e7fc700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 0x7f6a580b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:34.801 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.801+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f6a600e3830 con 0x7f6a7006b220 2024-05-26T12:01:34.802 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.801+0000 7f6a6e7fc700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 0x7f6a580b4e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6a64005950 tx=0x7f6a640058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:34.804 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:34.804+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6a60031070 con 0x7f6a7006b220 2024-05-26T12:01:35.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.014+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6a7019af20 con 0x7f6a7006b220 2024-05-26T12:01:35.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.015+0000 7f6a7499c700 1 -- 172.21.15.156:0/1366998827 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+22707 (secure 0 0 0) 0x7f6a60013070 con 0x7f6a7006b220 2024-05-26T12:01:35.016 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:01:35.016 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":81,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:01:07.937226+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:01:35.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 msgr2=0x7f6a580b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:35.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 0x7f6a580b4e50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6a64005950 tx=0x7f6a640058e0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 msgr2=0x7f6a7019f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6a60000c00 tx=0x7f6a600047b0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 shutdown_connections 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6a580b2990 0x7f6a580b4e50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a701095b0 0x7f6a70199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6a7006b220 0x7f6a7019f830 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 --2- 172.21.15.156:0/1366998827 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6a7006bbe0 0x7f6a7019fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 >> 172.21.15.156:0/1366998827 conn(0x7f6a700793f0 msgr2=0x7f6a70101140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 shutdown_connections 2024-05-26T12:01:35.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:35.018+0000 7f6a7599e700 1 -- 172.21.15.156:0/1366998827 wait complete. 2024-05-26T12:01:35.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:35 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1366998827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:35.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:35 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1366998827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:35.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:35 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1366998827' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:01:35.752 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-26T12:01:35.752 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-26T12:01:35.753 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-26T12:01:35.753 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-26T12:01:35.754 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-26T12:01:35.754 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-26T12:01:35.755 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-26T12:01:35.755 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-26T12:01:35.756 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-26T12:01:35.756 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-26T12:01:35.756 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-26T12:01:35.757 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-26T12:01:36.363 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:36 smithi186 ceph-mon[36412]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:36.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:36 smithi156 ceph-mon[32025]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:36.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:36 smithi192 ceph-mon[36545]: pgmap v325: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:36.807 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.809 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.810 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.812 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.814 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:36.825 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:37.263 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:38.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:38 smithi156 ceph-mon[32025]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:01:38.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:38 smithi186 ceph-mon[36412]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:01:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:38 smithi192 ceph-mon[36545]: pgmap v326: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:01:40.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:40 smithi186 ceph-mon[36412]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-05-26T12:01:40.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:40 smithi156 ceph-mon[32025]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-05-26T12:01:40.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:40 smithi192 ceph-mon[36545]: pgmap v327: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 8 op/s 2024-05-26T12:01:42.188 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:42 smithi156 ceph-mon[32025]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-26T12:01:42.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:42 smithi186 ceph-mon[36412]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-26T12:01:42.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:42 smithi192 ceph-mon[36545]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-05-26T12:01:44.157 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:44.267 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:44 smithi156 ceph-mon[32025]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-26T12:01:44.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:44 smithi186 ceph-mon[36412]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-26T12:01:44.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:44 smithi192 ceph-mon[36545]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-05-26T12:01:45.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 -- 172.21.15.156:0/2509801193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378108520 msgr2=0x7f7378108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 --2- 172.21.15.156:0/2509801193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378108520 0x7f7378108900 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f7368009990 tx=0x7f7368009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 -- 172.21.15.156:0/2509801193 shutdown_connections 2024-05-26T12:01:45.023 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 --2- 172.21.15.156:0/2509801193 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378102e00 0x7f737810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 --2- 172.21.15.156:0/2509801193 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378108520 0x7f7378108900 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 --2- 172.21.15.156:0/2509801193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7378102440 0x7f73781028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 -- 172.21.15.156:0/2509801193 >> 172.21.15.156:0/2509801193 conn(0x7f73780793f0 msgr2=0x7f7378079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:45.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 -- 172.21.15.156:0/2509801193 shutdown_connections 2024-05-26T12:01:45.024 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.022+0000 7f737cd12700 1 -- 172.21.15.156:0/2509801193 wait complete. 2024-05-26T12:01:45.025 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.023+0000 7f737cd12700 1 Processor -- start 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.023+0000 7f737cd12700 1 -- start start 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7378102440 0x7f7378104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 0x7f73781045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378108520 0x7f7378104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7378112650 con 0x7f7378102e00 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f73781124d0 con 0x7f7378102440 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737cd12700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f73781127d0 con 0x7f7378108520 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 0x7f73781045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 0x7f73781045c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54636/0 (socket says 172.21.15.156:54636) 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 -- 172.21.15.156:0/3373767959 learned_addr learned my addr 172.21.15.156:0/3373767959 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f737659c700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7378102440 0x7f7378104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.026 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f7376d9d700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378108520 0x7f7378104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 -- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378108520 msgr2=0x7f7378104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378108520 0x7f7378104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 -- 172.21.15.156:0/3373767959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7378102440 msgr2=0x7f7378104080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7378102440 0x7f7378104080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73781a8100 con 0x7f7378102e00 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f7376d9d700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7378108520 0x7f7378104b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736dd9b700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 0x7f73781045c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f736000c9b0 tx=0x7f736000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.024+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360007b00 con 0x7f7378102e00 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.025+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7360007ca0 con 0x7f7378102e00 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.025+0000 7f737cd12700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73781a83f0 con 0x7f7378102e00 2024-05-26T12:01:45.027 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.025+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7360018660 con 0x7f7378102e00 2024-05-26T12:01:45.028 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.025+0000 7f737cd12700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73781a8930 con 0x7f7378102e00 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.026+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f735c001030 con 0x7f7378102e00 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.027+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f736000f450 con 0x7f7378102e00 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f736ffff700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73640b2910 0x7f73640b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f737659c700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73640b2910 0x7f73640b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f73600e2800 con 0x7f7378102e00 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f736ffff700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] conn(0x7f73640bc4d0 0x7f73640be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7360015e70 con 0x7f73640bc4d0 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f73600e2c50 con 0x7f7378102e00 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f737659c700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73640b2910 0x7f73640b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f736800b5c0 tx=0x7f7368005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.029 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.028+0000 7f7376d9d700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] conn(0x7f73640bc4d0 0x7f73640be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.029+0000 7f7376d9d700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] conn(0x7f73640bc4d0 0x7f73640be970 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.030 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.030+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== osd.8 v2:172.21.15.192:6816/3218806357 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7360015e70 con 0x7f73640bc4d0 2024-05-26T12:01:45.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.046+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 --> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f735c002e60 con 0x7f73640bc4d0 2024-05-26T12:01:45.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.046+0000 7f736ffff700 1 -- 172.21.15.156:0/3373767959 <== osd.8 v2:172.21.15.192:6816/3218806357 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f735c002e60 con 0x7f73640bc4d0 2024-05-26T12:01:45.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.046+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] conn(0x7f73640bc4d0 msgr2=0x7f73640be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] conn(0x7f73640bc4d0 0x7f73640be970 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73640b2910 msgr2=0x7f73640b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f73640b2910 0x7f73640b4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f736800b5c0 tx=0x7f7368005960 comp rx=0 tx=0).stop 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 msgr2=0x7f73781045c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 --2- 172.21.15.156:0/3373767959 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7378102e00 0x7f73781045c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f736000c9b0 tx=0x7f736000ccc0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f7376d9d700 1 -- 172.21.15.156:0/3373767959 reap_dead start 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 shutdown_connections 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 >> 172.21.15.156:0/3373767959 conn(0x7f73780793f0 msgr2=0x7f737806b360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 shutdown_connections 2024-05-26T12:01:45.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:45.047+0000 7f736d59a700 1 -- 172.21.15.156:0/3373767959 wait complete. 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 -- 172.21.15.156:0/484356392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168108520 msgr2=0x7f8168108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 --2- 172.21.15.156:0/484356392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168108520 0x7f8168108900 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f8158009990 tx=0x7f8158009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 -- 172.21.15.156:0/484356392 shutdown_connections 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 --2- 172.21.15.156:0/484356392 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 0x7f816810f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 --2- 172.21.15.156:0/484356392 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168108520 0x7f8168108900 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 --2- 172.21.15.156:0/484356392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168102440 0x7f81681028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.455 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 -- 172.21.15.156:0/484356392 >> 172.21.15.156:0/484356392 conn(0x7f81680793f0 msgr2=0x7f8168079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 -- 172.21.15.156:0/484356392 shutdown_connections 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.016+0000 7f816c9de700 1 -- 172.21.15.156:0/484356392 wait complete. 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 Processor -- start 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 -- start start 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 0x7f8168104080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 0x7f81681045c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168108520 0x7f8168104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8168112650 con 0x7f8168102440 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f81681124d0 con 0x7f8168108520 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816c9de700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f81681127d0 con 0x7f8168102e00 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 0x7f8168104080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 0x7f8168104080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:46616/0 (socket says 172.21.15.156:46616) 2024-05-26T12:01:45.456 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 -- 172.21.15.156:0/923470328 learned_addr learned my addr 172.21.15.156:0/923470328 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f8166d9d700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168108520 0x7f8168104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f8165d9b700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 0x7f81681045c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 -- 172.21.15.156:0/923470328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 msgr2=0x7f81681045c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 0x7f81681045c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 -- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168108520 msgr2=0x7f8168104b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8168108520 0x7f8168104b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f816659c700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81681a8180 con 0x7f8168102440 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.017+0000 7f8165d9b700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8168102e00 0x7f81681045c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f816659c700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 0x7f8168104080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8158000c00 tx=0x7f81580047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8158027070 con 0x7f8168102440 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81681a8410 con 0x7f8168102440 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f81580040e0 con 0x7f8168102440 2024-05-26T12:01:45.457 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f815800f460 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.018+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81681a8860 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.019+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f81681074a0 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.020+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f815802a070 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.021+0000 7f81577fe700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f81500b2910 0x7f81500b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f8165d9b700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f81500b2910 0x7f81500b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f81580e3ad0 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f81577fe700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] conn(0x7f81500bc4d0 0x7f81500be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81500bf040 con 0x7f81500bc4d0 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f81580e3f20 con 0x7f8168102440 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f8165d9b700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f81500b2910 0x7f81500b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f815c005fd0 tx=0x7f815c005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.022+0000 7f8166d9d700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] conn(0x7f81500bc4d0 0x7f81500be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.023+0000 7f8166d9d700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] conn(0x7f81500bc4d0 0x7f81500be970 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:45.458 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.024+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== osd.4 v2:172.21.15.186:6808/1136966135 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f81500bf040 con 0x7f81500bc4d0 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.040+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 --> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8168103b60 con 0x7f81500bc4d0 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.040+0000 7f81577fe700 1 -- 172.21.15.156:0/923470328 <== osd.4 v2:172.21.15.186:6808/1136966135 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8168103b60 con 0x7f81500bc4d0 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] conn(0x7f81500bc4d0 msgr2=0x7f81500be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] conn(0x7f81500bc4d0 0x7f81500be970 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f81500b2910 msgr2=0x7f81500b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f81500b2910 0x7f81500b4dd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f815c005fd0 tx=0x7f815c005dc0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 msgr2=0x7f8168104080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 --2- 172.21.15.156:0/923470328 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8168102440 0x7f8168104080 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8158000c00 tx=0x7f81580047b0 comp rx=0 tx=0).stop 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f8166d9d700 1 -- 172.21.15.156:0/923470328 reap_dead start 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 shutdown_connections 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 >> 172.21.15.156:0/923470328 conn(0x7f81680793f0 msgr2=0x7f816806b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 shutdown_connections 2024-05-26T12:01:45.460 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:44.041+0000 7f816c9de700 1 -- 172.21.15.156:0/923470328 wait complete. 2024-05-26T12:01:45.662 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:01:46.091 INFO:teuthology.orchestra.run.smithi156.stdout:137438953539 2024-05-26T12:01:46.091 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-26T12:01:46.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:46 smithi186 ceph-mon[36412]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-26T12:01:46.495 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:46 smithi156 ceph-mon[32025]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-26T12:01:46.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:46 smithi192 ceph-mon[36545]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-05-26T12:01:46.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 -- 172.21.15.156:0/2412890414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01024d0 msgr2=0x7f50a0102950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 --2- 172.21.15.156:0/2412890414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01024d0 0x7f50a0102950 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f508c009990 tx=0x7f508c009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 -- 172.21.15.156:0/2412890414 shutdown_connections 2024-05-26T12:01:46.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 --2- 172.21.15.156:0/2412890414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 0x7f50a010f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 --2- 172.21.15.156:0/2412890414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a01084d0 0x7f50a01088b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 --2- 172.21.15.156:0/2412890414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01024d0 0x7f50a0102950 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.685+0000 7f50a709f700 1 -- 172.21.15.156:0/2412890414 >> 172.21.15.156:0/2412890414 conn(0x7f50a0078490 msgr2=0x7f50a00788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- 172.21.15.156:0/2412890414 shutdown_connections 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- 172.21.15.156:0/2412890414 wait complete. 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 Processor -- start 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- start start 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a01024d0 0x7f50a01040a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 0x7f50a01045e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 0x7f50a0104b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f50a01125e0 con 0x7f50a01084d0 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0112460 con 0x7f50a0102e90 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.686+0000 7f50a709f700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f50a0112760 con 0x7f50a01024d0 2024-05-26T12:01:46.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 0x7f50a0104b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 0x7f50a0104b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54658/0 (socket says 172.21.15.156:54658) 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 -- 172.21.15.156:0/2905778797 learned_addr learned my addr 172.21.15.156:0/2905778797 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f509ffff700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 0x7f50a01045e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a4e3b700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a01024d0 0x7f50a01040a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 -- 172.21.15.156:0/2905778797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a01024d0 msgr2=0x7f50a01040a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f50a01024d0 0x7f50a01040a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 -- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 msgr2=0x7f50a01045e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 0x7f50a01045e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50a01a80b0 con 0x7f50a01084d0 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f509ffff700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50a0102e90 0x7f50a01045e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a563c700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 0x7f50a0104b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5098007ee0 tx=0x7f509800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:46.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509800dd20 con 0x7f50a01084d0 2024-05-26T12:01:46.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50a01a83a0 con 0x7f50a01084d0 2024-05-26T12:01:46.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5098004020 con 0x7f50a01084d0 2024-05-26T12:01:46.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.687+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5098015660 con 0x7f50a01084d0 2024-05-26T12:01:46.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.688+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50a01a8860 con 0x7f50a01084d0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.688+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f50a0100610 con 0x7f50a01084d0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.690+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f50980041d0 con 0x7f50a01084d0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.690+0000 7f509dffb700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f50900b2910 0x7f50900b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f50a4e3b700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f50900b2910 0x7f50900b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f50980e28d0 con 0x7f50a01084d0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f509dffb700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] conn(0x7f50900bc4d0 0x7f50900be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f509800be50 con 0x7f50900bc4d0 2024-05-26T12:01:46.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f50980e2d20 con 0x7f50a01084d0 2024-05-26T12:01:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f50a4e3b700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f50900b2910 0x7f50900b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f509400abd0 tx=0x7f509400a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.691+0000 7f509ffff700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] conn(0x7f50900bc4d0 0x7f50900be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.692+0000 7f509ffff700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] conn(0x7f50900bc4d0 0x7f50900be970 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:46.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.692+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== osd.10 v2:172.21.15.186:6824/714032142 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f509800be50 con 0x7f50900bc4d0 2024-05-26T12:01:46.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.709+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 --> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f50a0103b80 con 0x7f50900bc4d0 2024-05-26T12:01:46.709 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.709+0000 7f509dffb700 1 -- 172.21.15.156:0/2905778797 <== osd.10 v2:172.21.15.186:6824/714032142 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f50a0103b80 con 0x7f50900bc4d0 2024-05-26T12:01:46.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] conn(0x7f50900bc4d0 msgr2=0x7f50900be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] conn(0x7f50900bc4d0 0x7f50900be970 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f50900b2910 msgr2=0x7f50900b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.710 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f50900b2910 0x7f50900b4dd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f509400abd0 tx=0x7f509400a380 comp rx=0 tx=0).stop 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 msgr2=0x7f50a0104b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 --2- 172.21.15.156:0/2905778797 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f50a01084d0 0x7f50a0104b20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5098007ee0 tx=0x7f509800d3b0 comp rx=0 tx=0).stop 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a563c700 1 -- 172.21.15.156:0/2905778797 reap_dead start 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 shutdown_connections 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 >> 172.21.15.156:0/2905778797 conn(0x7f50a0078490 msgr2=0x7f50a01062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 shutdown_connections 2024-05-26T12:01:46.711 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:46.710+0000 7f50a709f700 1 -- 172.21.15.156:0/2905778797 wait complete. 2024-05-26T12:01:48.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:48 smithi156 ceph-mon[32025]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-05-26T12:01:48.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:48 smithi186 ceph-mon[36412]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-05-26T12:01:48.509 INFO:teuthology.orchestra.run.smithi156.stdout:279172874276 2024-05-26T12:01:48.510 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-26T12:01:48.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:48 smithi192 ceph-mon[36545]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 10 op/s 2024-05-26T12:01:49.079 INFO:teuthology.orchestra.run.smithi156.stdout:231928234031 2024-05-26T12:01:49.079 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- 172.21.15.156:0/2944641615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8108510 msgr2=0x7f03f81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 --2- 172.21.15.156:0/2944641615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8108510 0x7f03f81088f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f03e8009990 tx=0x7f03e8009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- 172.21.15.156:0/2944641615 shutdown_connections 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 --2- 172.21.15.156:0/2944641615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f8102df0 0x7f03f810f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 --2- 172.21.15.156:0/2944641615 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8108510 0x7f03f81088f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 --2- 172.21.15.156:0/2944641615 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03f8102430 0x7f03f81028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- 172.21.15.156:0/2944641615 >> 172.21.15.156:0/2944641615 conn(0x7f03f80793b0 msgr2=0x7f03f80797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:49.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- 172.21.15.156:0/2944641615 shutdown_connections 2024-05-26T12:01:49.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- 172.21.15.156:0/2944641615 wait complete. 2024-05-26T12:01:49.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 Processor -- start 2024-05-26T12:01:49.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.763+0000 7f03fd897700 1 -- start start 2024-05-26T12:01:49.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 0x7f03f819f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f8102df0 0x7f03f819fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03f8108510 0x7f03f8199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f03f8112640 con 0x7f03f8102430 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f03f81124c0 con 0x7f03f8102df0 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03fd897700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f03f81127c0 con 0x7f03f8108510 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 0x7f03f819f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 0x7f03f819f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54664/0 (socket says 172.21.15.156:54664) 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 -- 172.21.15.156:0/2590156075 learned_addr learned my addr 172.21.15.156:0/2590156075 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03ee7fc700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f8102df0 0x7f03f819fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 -- 172.21.15.156:0/2590156075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03f8108510 msgr2=0x7f03f8199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f77fe700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03f8108510 0x7f03f8199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f03f8108510 0x7f03f8199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 -- 172.21.15.156:0/2590156075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f8102df0 msgr2=0x7f03f819fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f8102df0 0x7f03f819fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03f819a210 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.764+0000 7f03f6ffd700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 0x7f03f819f8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03e8000c00 tx=0x7f03e80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e8027070 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03e80040e0 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03f819a4a0 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e800f460 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03f81a67c0 con 0x7f03f8102430 2024-05-26T12:01:49.768 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.765+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f03f8107510 con 0x7f03f8102430 2024-05-26T12:01:49.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.767+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f03e802a070 con 0x7f03f8102430 2024-05-26T12:01:49.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03fc895700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f03e40b2910 0x7f03e40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:49.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03ee7fc700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f03e40b2910 0x7f03e40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:49.769 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f03e80e3a80 con 0x7f03f8102430 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03fc895700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] conn(0x7f03e40bc3a0 0x7f03e40be840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03e40bef10 con 0x7f03e40bc3a0 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f03e80e3ed0 con 0x7f03f8102430 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03f77fe700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] conn(0x7f03e40bc3a0 0x7f03e40be840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.768+0000 7f03ee7fc700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f03e40b2910 0x7f03e40b4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f03f8079920 tx=0x7f03e0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.769+0000 7f03f77fe700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] conn(0x7f03e40bc3a0 0x7f03e40be840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:49.770 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.769+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== osd.9 v2:172.21.15.156:6826/1302726213 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e40bef10 con 0x7f03e40bc3a0 2024-05-26T12:01:49.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.786+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 --> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f03f8103b50 con 0x7f03e40bc3a0 2024-05-26T12:01:49.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.786+0000 7f03fc895700 1 -- 172.21.15.156:0/2590156075 <== osd.9 v2:172.21.15.156:6826/1302726213 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f03f8103b50 con 0x7f03e40bc3a0 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.786+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] conn(0x7f03e40bc3a0 msgr2=0x7f03e40be840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.786+0000 7f03fd897700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] conn(0x7f03e40bc3a0 0x7f03e40be840 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f03e40b2910 msgr2=0x7f03e40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f03e40b2910 0x7f03e40b4dd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f03f8079920 tx=0x7f03e0009190 comp rx=0 tx=0).stop 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 msgr2=0x7f03f819f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 --2- 172.21.15.156:0/2590156075 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f03f8102430 0x7f03f819f8f0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03e8000c00 tx=0x7f03e80047b0 comp rx=0 tx=0).stop 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03f77fe700 1 -- 172.21.15.156:0/2590156075 reap_dead start 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 shutdown_connections 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 >> 172.21.15.156:0/2590156075 conn(0x7f03f80793b0 msgr2=0x7f03f806bc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 shutdown_connections 2024-05-26T12:01:49.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:49.787+0000 7f03fd897700 1 -- 172.21.15.156:0/2590156075 wait complete. 2024-05-26T12:01:50.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:50 smithi186 ceph-mon[36412]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:50.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:50 smithi156 ceph-mon[32025]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:50 smithi192 ceph-mon[36545]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:01:51.500 INFO:teuthology.orchestra.run.smithi156.stdout:257698037803 2024-05-26T12:01:51.501 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-26T12:01:52.441 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:52 smithi156 ceph-mon[32025]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-26T12:01:52.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:52 smithi186 ceph-mon[36412]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-26T12:01:52.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:52 smithi192 ceph-mon[36545]: pgmap v333: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 11 op/s 2024-05-26T12:01:54.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:54 smithi186 ceph-mon[36412]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:01:54.507 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:54 smithi156 ceph-mon[32025]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:01:54.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:54 smithi192 ceph-mon[36545]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:01:55.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 -- 172.21.15.156:0/4007094203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc108620 msgr2=0x7f36dc108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.158 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 --2- 172.21.15.156:0/4007094203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc108620 0x7f36dc108a00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f36cc009990 tx=0x7f36cc009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 -- 172.21.15.156:0/4007094203 shutdown_connections 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 --2- 172.21.15.156:0/4007094203 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36dc1028c0 0x7f36dc110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 --2- 172.21.15.156:0/4007094203 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc108620 0x7f36dc108a00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 --2- 172.21.15.156:0/4007094203 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36dc101f00 0x7f36dc102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 -- 172.21.15.156:0/4007094203 >> 172.21.15.156:0/4007094203 conn(0x7f36dc077780 msgr2=0x7f36dc077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 -- 172.21.15.156:0/4007094203 shutdown_connections 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.157+0000 7f36e13ee700 1 -- 172.21.15.156:0/4007094203 wait complete. 2024-05-26T12:01:55.159 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.158+0000 7f36e13ee700 1 Processor -- start 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.158+0000 7f36e13ee700 1 -- start start 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.158+0000 7f36e13ee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 0x7f36dc19f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.158+0000 7f36e13ee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36dc1028c0 0x7f36dc19fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36dc108620 0x7f36dc199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc113760 con 0x7f36dc101f00 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc1135e0 con 0x7f36dc1028c0 2024-05-26T12:01:55.161 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f36dc1138e0 con 0x7f36dc108620 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 0x7f36dc19f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 0x7f36dc19f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36168/0 (socket says 172.21.15.156:36168) 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 -- 172.21.15.156:0/1269419349 learned_addr learned my addr 172.21.15.156:0/1269419349 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36db7fe700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36dc108620 0x7f36dc199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36da7fc700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36dc1028c0 0x7f36dc19fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 -- 172.21.15.156:0/1269419349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36dc108620 msgr2=0x7f36dc199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f36dc108620 0x7f36dc199900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 -- 172.21.15.156:0/1269419349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36dc1028c0 msgr2=0x7f36dc19fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36dc1028c0 0x7f36dc19fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36dc19a060 con 0x7f36dc101f00 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36daffd700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 0x7f36dc19f740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc000c00 tx=0x7f36cc0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.162 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc027070 con 0x7f36dc101f00 2024-05-26T12:01:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36cc003fe0 con 0x7f36dc101f00 2024-05-26T12:01:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36dc19a2f0 con 0x7f36dc101f00 2024-05-26T12:01:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36cc022da0 con 0x7f36dc101f00 2024-05-26T12:01:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.159+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36dc1a6610 con 0x7f36dc101f00 2024-05-26T12:01:55.163 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.160+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f36dc101320 con 0x7f36dc101f00 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.162+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f36cc02a070 con 0x7f36dc101f00 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.162+0000 7f36c3fff700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f36c40b2910 0x7f36c40b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36da7fc700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f36c40b2910 0x7f36c40b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f36cc0e38f0 con 0x7f36dc101f00 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36c3fff700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] conn(0x7f36c40bc4d0 0x7f36c40be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36cc00eea0 con 0x7f36c40bc4d0 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f36cc0e3d40 con 0x7f36dc101f00 2024-05-26T12:01:55.164 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36db7fe700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] conn(0x7f36c40bc4d0 0x7f36c40be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36da7fc700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f36c40b2910 0x7f36c40b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36dc077c30 tx=0x7f36d000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.163+0000 7f36db7fe700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] conn(0x7f36c40bc4d0 0x7f36c40be970 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.164+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== osd.6 v2:172.21.15.156:6818/4035023489 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f36cc00eea0 con 0x7f36c40bc4d0 2024-05-26T12:01:55.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 --> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f36dc06b6f0 con 0x7f36c40bc4d0 2024-05-26T12:01:55.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36c3fff700 1 -- 172.21.15.156:0/1269419349 <== osd.6 v2:172.21.15.156:6818/4035023489 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f36dc06b6f0 con 0x7f36c40bc4d0 2024-05-26T12:01:55.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] conn(0x7f36c40bc4d0 msgr2=0x7f36c40be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] conn(0x7f36c40bc4d0 0x7f36c40be970 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f36c40b2910 msgr2=0x7f36c40b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f36c40b2910 0x7f36c40b4dd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f36dc077c30 tx=0x7f36d000a380 comp rx=0 tx=0).stop 2024-05-26T12:01:55.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 msgr2=0x7f36dc19f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 --2- 172.21.15.156:0/1269419349 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f36dc101f00 0x7f36dc19f740 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f36cc000c00 tx=0x7f36cc0047b0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36db7fe700 1 -- 172.21.15.156:0/1269419349 reap_dead start 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 shutdown_connections 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 >> 172.21.15.156:0/1269419349 conn(0x7f36dc077780 msgr2=0x7f36dc1008c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.181+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 shutdown_connections 2024-05-26T12:01:55.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.182+0000 7f36e13ee700 1 -- 172.21.15.156:0/1269419349 wait complete. 2024-05-26T12:01:55.258 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 -- 172.21.15.156:0/3936246508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541088f0 msgr2=0x7fd154110470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 --2- 172.21.15.156:0/3936246508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541088f0 0x7fd154110470 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd14c00b0d0 tx=0x7fd14c00b3e0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 -- 172.21.15.156:0/3936246508 shutdown_connections 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 --2- 172.21.15.156:0/3936246508 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541088f0 0x7fd154110470 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 --2- 172.21.15.156:0/3936246508 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1541037f0 0x7fd154103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 --2- 172.21.15.156:0/3936246508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 0x7fd1541081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 -- 172.21.15.156:0/3936246508 >> 172.21.15.156:0/3936246508 conn(0x7fd15407a6a0 msgr2=0x7fd15407aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 -- 172.21.15.156:0/3936246508 shutdown_connections 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.029+0000 7fd159c47700 1 -- 172.21.15.156:0/3936246508 wait complete. 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 Processor -- start 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 -- start start 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 0x7fd15419f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 0x7fd15419fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.259 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1541088f0 0x7fd154199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd154113740 con 0x7fd1541037f0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541135c0 con 0x7fd1541041a0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.030+0000 7fd159c47700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd1541138c0 con 0x7fd1541088f0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 0x7fd15419f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 0x7fd15419f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36158/0 (socket says 172.21.15.156:36158) 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 -- 172.21.15.156:0/3446187041 learned_addr learned my addr 172.21.15.156:0/3446187041 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd153fff700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1541088f0 0x7fd154199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd152ffd700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 0x7fd15419fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 -- 172.21.15.156:0/3446187041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1541088f0 msgr2=0x7fd154199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd1541088f0 0x7fd154199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 -- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 msgr2=0x7fd15419fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 0x7fd15419fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd15419a160 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd152ffd700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1541041a0 0x7fd15419fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd1537fe700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 0x7fd15419f7f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd14400b530 tx=0x7fd14400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144015020 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd15419a450 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd144011ce0 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd144010e40 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.031+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1541a66c0 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.032+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd154063cd0 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.034+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fd144018020 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.034+0000 7fd150ff9700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd13c0b2910 0x7fd13c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.034+0000 7fd152ffd700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd13c0b2910 0x7fd13c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.034+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd1440e2270 con 0x7fd1541037f0 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd150ff9700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] conn(0x7fd13c0bc4d0 0x7fd13c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd14401be50 con 0x7fd13c0bc4d0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fd1440e26c0 con 0x7fd1541037f0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd152ffd700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd13c0b2910 0x7fd13c0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd148005950 tx=0x7fd1480058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd153fff700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] conn(0x7fd13c0bc4d0 0x7fd13c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.035+0000 7fd153fff700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] conn(0x7fd13c0bc4d0 0x7fd13c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.036+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== osd.1 v2:172.21.15.186:6800/1098664623 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd14401be50 con 0x7fd13c0bc4d0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.053+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 --> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd154064150 con 0x7fd13c0bc4d0 2024-05-26T12:01:55.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.053+0000 7fd150ff9700 1 -- 172.21.15.156:0/3446187041 <== osd.1 v2:172.21.15.186:6800/1098664623 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fd154064150 con 0x7fd13c0bc4d0 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] conn(0x7fd13c0bc4d0 msgr2=0x7fd13c0be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] conn(0x7fd13c0bc4d0 0x7fd13c0be970 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd13c0b2910 msgr2=0x7fd13c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd13c0b2910 0x7fd13c0b4dd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd148005950 tx=0x7fd1480058e0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 msgr2=0x7fd15419f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 --2- 172.21.15.156:0/3446187041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd1541037f0 0x7fd15419f7f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fd14400b530 tx=0x7fd14400b840 comp rx=0 tx=0).stop 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd153fff700 1 -- 172.21.15.156:0/3446187041 reap_dead start 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 shutdown_connections 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 >> 172.21.15.156:0/3446187041 conn(0x7fd15407a6a0 msgr2=0x7fd154101770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 shutdown_connections 2024-05-26T12:01:55.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:55.054+0000 7fd159c47700 1 -- 172.21.15.156:0/3446187041 wait complete. 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 -- 172.21.15.156:0/1926367242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0108520 msgr2=0x7fd4b0108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/1926367242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0108520 0x7fd4b0108900 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac009990 tx=0x7fd4ac009ca0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 -- 172.21.15.156:0/1926367242 shutdown_connections 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/1926367242 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0102e00 0x7fd4b010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/1926367242 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0108520 0x7fd4b0108900 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/1926367242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0102440 0x7fd4b01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 -- 172.21.15.156:0/1926367242 >> 172.21.15.156:0/1926367242 conn(0x7fd4b00793f0 msgr2=0x7fd4b0079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 -- 172.21.15.156:0/1926367242 shutdown_connections 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.175+0000 7fd4b86f2700 1 -- 172.21.15.156:0/1926367242 wait complete. 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 Processor -- start 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 -- start start 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 0x7fd4b019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0102e00 0x7fd4b019fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0108520 0x7fd4b0199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b0112640 con 0x7fd4b0102440 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b01124c0 con 0x7fd4b0108520 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b86f2700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd4b01127c0 con 0x7fd4b0102e00 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 0x7fd4b019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 0x7fd4b019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:54684/0 (socket says 172.21.15.156:54684) 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 -- 172.21.15.156:0/99121200 learned_addr learned my addr 172.21.15.156:0/99121200 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b5c8d700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0102e00 0x7fd4b019fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b6c8f700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0108520 0x7fd4b0199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 -- 172.21.15.156:0/99121200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0102e00 msgr2=0x7fd4b019fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd4b0102e00 0x7fd4b019fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 -- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0108520 msgr2=0x7fd4b0199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0108520 0x7fd4b0199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b648e700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4b019a180 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.176+0000 7fd4b6c8f700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4b0108520 0x7fd4b0199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4b648e700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 0x7fd4b019f860 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005910 tx=0x7fd4ac00b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac027070 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4b019a410 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd4ac022470 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4ac017650 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.177+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4b01a6730 con 0x7fd4b0102440 2024-05-26T12:01:55.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.178+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fd4b01074e0 con 0x7fd4b0102440 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.179+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fd4ac02a070 con 0x7fd4b0102440 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4a77fe700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd49c0b2910 0x7fd49c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4b5c8d700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd49c0b2910 0x7fd49c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fd4ac0e3ae0 con 0x7fd4b0102440 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4a77fe700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] conn(0x7fd49c0bc4d0 0x7fd49c0be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd4ac007e80 con 0x7fd49c0bc4d0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fd4ac0e3f30 con 0x7fd4b0102440 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.180+0000 7fd4b5c8d700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd49c0b2910 0x7fd49c0b4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005ea0 tx=0x7fd4a0005e30 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.181+0000 7fd4b6c8f700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] conn(0x7fd49c0bc4d0 0x7fd49c0be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.181+0000 7fd4b6c8f700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] conn(0x7fd49c0bc4d0 0x7fd49c0be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.182+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== osd.7 v2:172.21.15.186:6816/2634988090 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fd4ac007e80 con 0x7fd49c0bc4d0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.199+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 --> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fd4b0103b60 con 0x7fd49c0bc4d0 2024-05-26T12:01:55.615 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.199+0000 7fd4a77fe700 1 -- 172.21.15.156:0/99121200 <== osd.7 v2:172.21.15.186:6816/2634988090 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fd4b0103b60 con 0x7fd49c0bc4d0 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] conn(0x7fd49c0bc4d0 msgr2=0x7fd49c0be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] conn(0x7fd49c0bc4d0 0x7fd49c0be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd49c0b2910 msgr2=0x7fd49c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd49c0b2910 0x7fd49c0b4dd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4a0005ea0 tx=0x7fd4a0005e30 comp rx=0 tx=0).stop 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 msgr2=0x7fd4b019f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 --2- 172.21.15.156:0/99121200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd4b0102440 0x7fd4b019f860 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd4ac005910 tx=0x7fd4ac00b670 comp rx=0 tx=0).stop 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b6c8f700 1 -- 172.21.15.156:0/99121200 reap_dead start 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 shutdown_connections 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 >> 172.21.15.156:0/99121200 conn(0x7fd4b00793f0 msgr2=0x7fd4b006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:55.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 shutdown_connections 2024-05-26T12:01:55.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:54.200+0000 7fd4b86f2700 1 -- 172.21.15.156:0/99121200 wait complete. 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- 172.21.15.156:0/3773186339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf01024f0 msgr2=0x7f3cf01103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/3773186339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf01024f0 0x7f3cf01103d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f3ce800b0d0 tx=0x7f3ce800b3e0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- 172.21.15.156:0/3773186339 shutdown_connections 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/3773186339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf01024f0 0x7f3cf01103d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/3773186339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf0109520 0x7f3cf0109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/3773186339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf0101b30 0x7f3cf0101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- 172.21.15.156:0/3773186339 >> 172.21.15.156:0/3773186339 conn(0x7f3cf0078ab0 msgr2=0x7f3cf00ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- 172.21.15.156:0/3773186339 shutdown_connections 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- 172.21.15.156:0/3773186339 wait complete. 2024-05-26T12:01:56.179 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 Processor -- start 2024-05-26T12:01:56.180 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.178+0000 7f3cf70a9700 1 -- start start 2024-05-26T12:01:56.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf0101b30 0x7f3cf0102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:56.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf01024f0 0x7f3cf0103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:56.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 0x7f3cf0103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:56.181 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf0113760 con 0x7f3cf0109520 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf01135e0 con 0x7f3cf01024f0 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3cf01138e0 con 0x7f3cf0101b30 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 0x7f3cf0103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 0x7f3cf0103790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36184/0 (socket says 172.21.15.156:36184) 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 -- 172.21.15.156:0/2394824160 learned_addr learned my addr 172.21.15.156:0/2394824160 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3ceffff700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf01024f0 0x7f3cf0103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf4e45700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf0101b30 0x7f3cf0102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 -- 172.21.15.156:0/2394824160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf0101b30 msgr2=0x7f3cf0102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3cf0101b30 0x7f3cf0102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 -- 172.21.15.156:0/2394824160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf01024f0 msgr2=0x7f3cf0103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf01024f0 0x7f3cf0103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cf01a8100 con 0x7f3cf0109520 2024-05-26T12:01:56.182 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3ceffff700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3cf01024f0 0x7f3cf0103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf5646700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 0x7f3cf0103790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009850 tx=0x7f3ce8007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce801f070 con 0x7f3cf0109520 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cf01a8390 con 0x7f3cf0109520 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.179+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ce8005520 con 0x7f3cf0109520 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.180+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ce801b870 con 0x7f3cf0109520 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.180+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cf01a87e0 con 0x7f3cf0109520 2024-05-26T12:01:56.183 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.180+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3cf0063c50 con 0x7f3cf0109520 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.182+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f3ce8022070 con 0x7f3cf0109520 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cedffb700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3ce00b2910 0x7f3ce00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cf4e45700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3ce00b2910 0x7f3ce00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f3ce800f020 con 0x7f3cf0109520 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cedffb700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] conn(0x7f3ce00bc4d0 0x7f3ce00be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ce8008ec0 con 0x7f3ce00bc4d0 2024-05-26T12:01:56.184 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f3ce80e3cc0 con 0x7f3cf0109520 2024-05-26T12:01:56.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3ceffff700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] conn(0x7f3ce00bc4d0 0x7f3ce00be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:56.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3cf4e45700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3ce00b2910 0x7f3ce00b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009bd0 tx=0x7f3ce4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:56.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.183+0000 7f3ceffff700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] conn(0x7f3ce00bc4d0 0x7f3ce00be970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:56.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.184+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== osd.0 v2:172.21.15.156:6802/728684380 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ce8008ec0 con 0x7f3ce00bc4d0 2024-05-26T12:01:56.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 --> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3cf00640d0 con 0x7f3ce00bc4d0 2024-05-26T12:01:56.201 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cedffb700 1 -- 172.21.15.156:0/2394824160 <== osd.0 v2:172.21.15.156:6802/728684380 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3cf00640d0 con 0x7f3ce00bc4d0 2024-05-26T12:01:56.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] conn(0x7f3ce00bc4d0 msgr2=0x7f3ce00be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] conn(0x7f3ce00bc4d0 0x7f3ce00be970 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3ce00b2910 msgr2=0x7f3ce00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3ce00b2910 0x7f3ce00b4dd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4009bd0 tx=0x7f3ce4009260 comp rx=0 tx=0).stop 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 msgr2=0x7f3cf0103790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf70a9700 1 --2- 172.21.15.156:0/2394824160 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3cf0109520 0x7f3cf0103790 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3ce8009850 tx=0x7f3ce8007ad0 comp rx=0 tx=0).stop 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.201+0000 7f3cf5646700 1 -- 172.21.15.156:0/2394824160 reap_dead start 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.202+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 shutdown_connections 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.202+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 >> 172.21.15.156:0/2394824160 conn(0x7f3cf0078ab0 msgr2=0x7f3cf0100220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.202+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 shutdown_connections 2024-05-26T12:01:56.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:56.202+0000 7f3cf70a9700 1 -- 172.21.15.156:0/2394824160 wait complete. 2024-05-26T12:01:56.392 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:56 smithi156 ceph-mon[32025]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-26T12:01:56.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:56 smithi186 ceph-mon[36412]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-26T12:01:56.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:56 smithi192 ceph-mon[36545]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-05-26T12:01:57.031 INFO:teuthology.orchestra.run.smithi156.stdout:34359738456 2024-05-26T12:01:57.031 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.490+0000 7f973d85d700 1 -- 172.21.15.156:0/3190232223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381041a0 msgr2=0x7f97381081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.490+0000 7f973d85d700 1 --2- 172.21.15.156:0/3190232223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381041a0 0x7f97381081c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f972c0099c0 tx=0x7f972c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 -- 172.21.15.156:0/3190232223 shutdown_connections 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 --2- 172.21.15.156:0/3190232223 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97381088f0 0x7f9738110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 --2- 172.21.15.156:0/3190232223 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97381037f0 0x7f9738103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 --2- 172.21.15.156:0/3190232223 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381041a0 0x7f97381081c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 -- 172.21.15.156:0/3190232223 >> 172.21.15.156:0/3190232223 conn(0x7f973807a6a0 msgr2=0x7f973807aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 -- 172.21.15.156:0/3190232223 shutdown_connections 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 -- 172.21.15.156:0/3190232223 wait complete. 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 Processor -- start 2024-05-26T12:01:57.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.491+0000 7f973d85d700 1 -- start start 2024-05-26T12:01:57.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 0x7f9738075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97381041a0 0x7f9738075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97381088f0 0x7f97380762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f9738113720 con 0x7f97381037f0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f97381135a0 con 0x7f97381041a0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f97381138a0 con 0x7f97381088f0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 0x7f9738075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 0x7f9738075840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36214/0 (socket says 172.21.15.156:36214) 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 -- 172.21.15.156:0/1186896320 learned_addr learned my addr 172.21.15.156:0/1186896320 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f97367fc700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97381041a0 0x7f9738075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f97377fe700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97381088f0 0x7f97380762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 -- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97381088f0 msgr2=0x7f97380762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f97381088f0 0x7f97380762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 -- 172.21.15.156:0/1186896320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97381041a0 msgr2=0x7f9738075d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97381041a0 0x7f9738075d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97381a8160 con 0x7f97381037f0 2024-05-26T12:01:57.495 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f9736ffd700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 0x7f9738075840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f972800d980 tx=0x7f972800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728017070 con 0x7f97381037f0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9728005020 con 0x7f97381037f0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97381a8450 con 0x7f97381037f0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9728009b50 con 0x7f97381037f0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.492+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97381a8910 con 0x7f97381037f0 2024-05-26T12:01:57.496 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.493+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9738063cd0 con 0x7f97381037f0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.495+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f97280051d0 con 0x7f97381037f0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.495+0000 7f973c85b700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f97200b2910 0x7f97200b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f97367fc700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f97200b2910 0x7f97200b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f97280e28b0 con 0x7f97381037f0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f973c85b700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] conn(0x7f97200bc4d0 0x7f97200be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97200bf040 con 0x7f97200bc4d0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f97280e2d00 con 0x7f97381037f0 2024-05-26T12:01:57.497 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f97367fc700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f97200b2910 0x7f97200b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b580 tx=0x7f972c005a90 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f97377fe700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] conn(0x7f97200bc4d0 0x7f97200be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.496+0000 7f97377fe700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] conn(0x7f97200bc4d0 0x7f97200be970 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.498 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.497+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== osd.2 v2:172.21.15.192:6800/4249467329 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f97200bf040 con 0x7f97200bc4d0 2024-05-26T12:01:57.514 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.514+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 --> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f9738064150 con 0x7f97200bc4d0 2024-05-26T12:01:57.515 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.514+0000 7f973c85b700 1 -- 172.21.15.156:0/1186896320 <== osd.2 v2:172.21.15.192:6800/4249467329 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f9738064150 con 0x7f97200bc4d0 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] conn(0x7f97200bc4d0 msgr2=0x7f97200be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] conn(0x7f97200bc4d0 0x7f97200be970 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f97200b2910 msgr2=0x7f97200b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f97200b2910 0x7f97200b4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f972c00b580 tx=0x7f972c005a90 comp rx=0 tx=0).stop 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 msgr2=0x7f9738075840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.516 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 --2- 172.21.15.156:0/1186896320 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f97381037f0 0x7f9738075840 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f972800d980 tx=0x7f972800dc90 comp rx=0 tx=0).stop 2024-05-26T12:01:57.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f97377fe700 1 -- 172.21.15.156:0/1186896320 reap_dead start 2024-05-26T12:01:57.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 shutdown_connections 2024-05-26T12:01:57.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 >> 172.21.15.156:0/1186896320 conn(0x7f973807a6a0 msgr2=0x7f97381019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:57.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 shutdown_connections 2024-05-26T12:01:57.517 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.515+0000 7f973d85d700 1 -- 172.21.15.156:0/1186896320 wait complete. 2024-05-26T12:01:57.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 -- 172.21.15.156:0/3629283918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c1042f0 msgr2=0x7fea6c1046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.746 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 --2- 172.21.15.156:0/3629283918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c1042f0 0x7fea6c1046d0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00b0a0 tx=0x7fea5c00b3b0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 -- 172.21.15.156:0/3629283918 shutdown_connections 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 --2- 172.21.15.156:0/3629283918 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea6c102e30 0x7fea6c10f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 --2- 172.21.15.156:0/3629283918 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c1042f0 0x7fea6c1046d0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 --2- 172.21.15.156:0/3629283918 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea6c102470 0x7fea6c1028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 -- 172.21.15.156:0/3629283918 >> 172.21.15.156:0/3629283918 conn(0x7fea6c0784b0 msgr2=0x7fea6c0788c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 -- 172.21.15.156:0/3629283918 shutdown_connections 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.740+0000 7fea72498700 1 -- 172.21.15.156:0/3629283918 wait complete. 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 Processor -- start 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 -- start start 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 0x7fea6c19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.747 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea6c102e30 0x7fea6c19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea6c1042f0 0x7fea6c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c1126f0 con 0x7fea6c102470 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c112570 con 0x7fea6c1042f0 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea72498700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fea6c112870 con 0x7fea6c102e30 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 0x7fea6c19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 0x7fea6c19e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36228/0 (socket says 172.21.15.156:36228) 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 -- 172.21.15.156:0/1246979913 learned_addr learned my addr 172.21.15.156:0/1246979913 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6b7fe700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea6c102e30 0x7fea6c19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 -- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea6c102e30 msgr2=0x7fea6c19ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fea6c102e30 0x7fea6c19ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 -- 172.21.15.156:0/1246979913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea6c1042f0 msgr2=0x7fea6c198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea6c1042f0 0x7fea6c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.748 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6c1990d0 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.741+0000 7fea6bfff700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 0x7fea6c19e7b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00b070 tx=0x7fea5c004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c021070 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea5c007c80 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6c199360 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5c01d450 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6c1a5680 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.742+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fea6c063cd0 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.744+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fea5c004000 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea697fa700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fea540b2990 0x7fea540b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea6b7fe700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fea540b2990 0x7fea540b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7fea5c00f020 con 0x7fea6c102470 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea697fa700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] conn(0x7fea540bc550 0x7fea540be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fea5c026e50 con 0x7fea540bc550 2024-05-26T12:01:57.749 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7fea5c0e3fa0 con 0x7fea6c102470 2024-05-26T12:01:57.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea6b7fe700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fea540b2990 0x7fea540b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0789b0 tx=0x7fea6000b5b0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.745+0000 7fea70a35700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] conn(0x7fea540bc550 0x7fea540be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:01:57.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.746+0000 7fea70a35700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] conn(0x7fea540bc550 0x7fea540be9f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:01:57.750 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.747+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== osd.5 v2:172.21.15.192:6808/199714532 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fea5c026e50 con 0x7fea540bc550 2024-05-26T12:01:57.764 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.764+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 --> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fea6c064100 con 0x7fea540bc550 2024-05-26T12:01:57.765 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea697fa700 1 -- 172.21.15.156:0/1246979913 <== osd.5 v2:172.21.15.192:6808/199714532 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fea6c064100 con 0x7fea540bc550 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] conn(0x7fea540bc550 msgr2=0x7fea540be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] conn(0x7fea540bc550 0x7fea540be9f0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fea540b2990 msgr2=0x7fea540b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fea540b2990 0x7fea540b4e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fea6c0789b0 tx=0x7fea6000b5b0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 msgr2=0x7fea6c19e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 --2- 172.21.15.156:0/1246979913 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fea6c102470 0x7fea6c19e7b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fea5c00b070 tx=0x7fea5c004db0 comp rx=0 tx=0).stop 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea70a35700 1 -- 172.21.15.156:0/1246979913 reap_dead start 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 shutdown_connections 2024-05-26T12:01:57.766 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.765+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 >> 172.21.15.156:0/1246979913 conn(0x7fea6c0784b0 msgr2=0x7fea6c107260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:01:57.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.766+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 shutdown_connections 2024-05-26T12:01:57.767 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:01:57.766+0000 7fea72498700 1 -- 172.21.15.156:0/1246979913 wait complete. 2024-05-26T12:01:58.455 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:01:58 smithi156 ceph-mon[32025]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:01:58.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:01:58 smithi186 ceph-mon[36412]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:01:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:01:58 smithi192 ceph-mon[36545]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:01:59.134 INFO:teuthology.orchestra.run.smithi156.stdout:184683593787 2024-05-26T12:01:59.134 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-26T12:01:59.707 INFO:teuthology.orchestra.run.smithi156.stdout:77309411408 2024-05-26T12:01:59.708 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-26T12:02:00.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:00 smithi186 ceph-mon[36412]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-26T12:02:00.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:00 smithi156 ceph-mon[32025]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-26T12:02:00.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:00 smithi192 ceph-mon[36545]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-05-26T12:02:00.694 INFO:teuthology.orchestra.run.smithi156.stdout:55834574932 2024-05-26T12:02:00.695 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-26T12:02:01.086 INFO:teuthology.orchestra.run.smithi156.stdout:158913790016 2024-05-26T12:02:01.087 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-26T12:02:01.473 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:02.209 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:02.209 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3705857663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c1095a0 msgr2=0x7f7c1c109980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3705857663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c1095a0 0x7f7c1c109980 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009990 tx=0x7f7c0c009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3705857663 shutdown_connections 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3705857663 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c1c06c470 0x7f7c1c1103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3705857663 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c1095a0 0x7f7c1c109980 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3705857663 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c06bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.210 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.209+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3705857663 >> 172.21.15.156:0/3705857663 conn(0x7f7c1c0793b0 msgr2=0x7f7c1c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:02.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.210+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3705857663 shutdown_connections 2024-05-26T12:02:02.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.210+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3705857663 wait complete. 2024-05-26T12:02:02.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.210+0000 7f7c22eb7700 1 Processor -- start 2024-05-26T12:02:02.211 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 -- start start 2024-05-26T12:02:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c10f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c1c06c470 0x7f7c1c10fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c1c1095a0 0x7f7c1c1a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c20c53700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c10f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:02.212 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c20c53700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c10f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36256/0 (socket says 172.21.15.156:36256) 2024-05-26T12:02:02.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c20c53700 1 -- 172.21.15.156:0/3926719098 learned_addr learned my addr 172.21.15.156:0/3926719098 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:02.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c1136b0 con 0x7f7c1c06bab0 2024-05-26T12:02:02.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c113530 con 0x7f7c1c1095a0 2024-05-26T12:02:02.213 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7c1c113830 con 0x7f7c1c06c470 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c1bfff700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c1c06c470 0x7f7c1c10fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.211+0000 7f7c21454700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c1c1095a0 0x7f7c1c1a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 -- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c1c06c470 msgr2=0x7f7c1c10fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7c1c06c470 0x7f7c1c10fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 -- 172.21.15.156:0/3926719098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c1c1095a0 msgr2=0x7f7c1c1a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c1c1095a0 0x7f7c1c1a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.214 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c1c1a51c0 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c20c53700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c10f750 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0057c0 tx=0x7f7c0c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c026070 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7c0c021410 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.212+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c017650 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.213+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c1c1a5450 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.214+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c1c1a58e0 con 0x7f7c1c06bab0 2024-05-26T12:02:02.215 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.215+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7c1c101a70 con 0x7f7c1c06bab0 2024-05-26T12:02:02.218 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.218+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f7c0c029070 con 0x7f7c1c06bab0 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.219+0000 7f7c19ffb700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c080b2990 0x7f7c080b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.219+0000 7f7c1bfff700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c080b2990 0x7f7c080b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.219+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f7c0c0e3620 con 0x7f7c1c06bab0 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c19ffb700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] conn(0x7f7c080bc550 0x7f7c080be9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c0c001d30 con 0x7f7c080bc550 2024-05-26T12:02:02.220 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f7c0c0e3a70 con 0x7f7c1c06bab0 2024-05-26T12:02:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c21454700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] conn(0x7f7c080bc550 0x7f7c080be9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c1bfff700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c080b2990 0x7f7c080b4e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c104520 tx=0x7f7c040073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:02.221 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.220+0000 7f7c21454700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] conn(0x7f7c080bc550 0x7f7c080be9f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:02.222 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.221+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== osd.11 v2:172.21.15.192:6824/345136915 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7c0c001d30 con 0x7f7c080bc550 2024-05-26T12:02:02.248 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.248+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 --> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7c1c104c60 con 0x7f7c080bc550 2024-05-26T12:02:02.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c19ffb700 1 -- 172.21.15.156:0/3926719098 <== osd.11 v2:172.21.15.192:6824/345136915 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7c1c104c60 con 0x7f7c080bc550 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] conn(0x7f7c080bc550 msgr2=0x7f7c080be9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] conn(0x7f7c080bc550 0x7f7c080be9f0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c080b2990 msgr2=0x7f7c080b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7c080b2990 0x7f7c080b4e50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7c1c104520 tx=0x7f7c040073d0 comp rx=0 tx=0).stop 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 msgr2=0x7f7c1c10f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 --2- 172.21.15.156:0/3926719098 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7c1c06bab0 0x7f7c1c10f750 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c0057c0 tx=0x7f7c0c009580 comp rx=0 tx=0).stop 2024-05-26T12:02:02.250 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c21454700 1 -- 172.21.15.156:0/3926719098 reap_dead start 2024-05-26T12:02:02.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 shutdown_connections 2024-05-26T12:02:02.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 >> 172.21.15.156:0/3926719098 conn(0x7f7c1c0793b0 msgr2=0x7f7c1c101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:02.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 shutdown_connections 2024-05-26T12:02:02.251 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.249+0000 7f7c22eb7700 1 -- 172.21.15.156:0/3926719098 wait complete. 2024-05-26T12:02:02.286 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:02 smithi156 ceph-mon[32025]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-26T12:02:02.479 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:02 smithi186 ceph-mon[36412]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-26T12:02:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:02 smithi192 ceph-mon[36545]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-05-26T12:02:03.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 -- 172.21.15.156:0/3737955551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4954109620 msgr2=0x7f4954109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.739 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 --2- 172.21.15.156:0/3737955551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4954109620 0x7f4954109a00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f4948009990 tx=0x7f4948009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 -- 172.21.15.156:0/3737955551 shutdown_connections 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 --2- 172.21.15.156:0/3737955551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f495406c670 0x7f4954110410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 --2- 172.21.15.156:0/3737955551 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4954109620 0x7f4954109a00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 --2- 172.21.15.156:0/3737955551 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f495406bcb0 0x7f495406c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.396+0000 7f495adc5700 1 -- 172.21.15.156:0/3737955551 >> 172.21.15.156:0/3737955551 conn(0x7f4954078490 msgr2=0x7f49540788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- 172.21.15.156:0/3737955551 shutdown_connections 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- 172.21.15.156:0/3737955551 wait complete. 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 Processor -- start 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- start start 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f495406bcb0 0x7f4954199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:03.740 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 0x7f495419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4954109620 0x7f495419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4954113800 con 0x7f495406c670 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4954113680 con 0x7f4954109620 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f495adc5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4954113980 con 0x7f495406bcb0 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f4953fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 0x7f495419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f4953fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 0x7f495419a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36266/0 (socket says 172.21.15.156:36266) 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.397+0000 7f4953fff700 1 -- 172.21.15.156:0/375446080 learned_addr learned my addr 172.21.15.156:0/375446080 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4958b61700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f495406bcb0 0x7f4954199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:03.741 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4959362700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4954109620 0x7f495419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 -- 172.21.15.156:0/375446080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f495406bcb0 msgr2=0x7f4954199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f495406bcb0 0x7f4954199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 -- 172.21.15.156:0/375446080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4954109620 msgr2=0x7f495419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4954109620 0x7f495419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f495419ed80 con 0x7f495406c670 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4953fff700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 0x7f495419a2b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f494000b530 tx=0x7f494000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940015020 con 0x7f495406c670 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f495419f070 con 0x7f495406c670 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4940011ce0 con 0x7f495406c670 2024-05-26T12:02:03.742 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4940010e40 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.398+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49541a68f0 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.399+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4954063c90 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.401+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f4940018020 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.401+0000 7f4951ffb700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49440b2910 0x7f49440b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.401+0000 7f4958b61700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49440b2910 0x7f49440b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.401+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8248+0+0 (secure 0 0 0) 0x7f49400e2250 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4951ffb700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] conn(0x7f49440bc4d0 0x7f49440be970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f494001be50 con 0x7f49440bc4d0 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_get_version_reply(handle=1 version=81) ==== 24+0+0 (secure 0 0 0) 0x7f49400e26a0 con 0x7f495406c670 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4959362700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] conn(0x7f49440bc4d0 0x7f49440be970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4958b61700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49440b2910 0x7f49440b4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f49480098e0 tx=0x7f4948005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.402+0000 7f4959362700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] conn(0x7f49440bc4d0 0x7f49440be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:03.743 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.403+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== osd.3 v2:172.21.15.156:6810/2032209985 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f494001be50 con 0x7f49440bc4d0 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.419+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 --> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f49540640c0 con 0x7f49440bc4d0 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.419+0000 7f4951ffb700 1 -- 172.21.15.156:0/375446080 <== osd.3 v2:172.21.15.156:6810/2032209985 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f49540640c0 con 0x7f49440bc4d0 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] conn(0x7f49440bc4d0 msgr2=0x7f49440be970 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] conn(0x7f49440bc4d0 0x7f49440be970 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49440b2910 msgr2=0x7f49440b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f49440b2910 0x7f49440b4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f49480098e0 tx=0x7f4948005960 comp rx=0 tx=0).stop 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 msgr2=0x7f495419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 --2- 172.21.15.156:0/375446080 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f495406c670 0x7f495419a2b0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f494000b530 tx=0x7f494000b840 comp rx=0 tx=0).stop 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f4959362700 1 -- 172.21.15.156:0/375446080 reap_dead start 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 shutdown_connections 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 >> 172.21.15.156:0/375446080 conn(0x7f4954078490 msgr2=0x7f4954101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 shutdown_connections 2024-05-26T12:02:03.744 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:02.420+0000 7f495adc5700 1 -- 172.21.15.156:0/375446080 wait complete. 2024-05-26T12:02:04.477 INFO:teuthology.orchestra.run.smithi156.stdout:111669149772 2024-05-26T12:02:04.478 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-26T12:02:04.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:04 smithi156 ceph-mon[32025]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:04.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:04 smithi192 ceph-mon[36545]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:04.926 INFO:teuthology.orchestra.run.smithi156.stdout:206158430262 2024-05-26T12:02:04.926 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-26T12:02:04.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:04 smithi186 ceph-mon[36412]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:06.128 INFO:teuthology.orchestra.run.smithi156.stdout:304942678050 2024-05-26T12:02:06.128 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-26T12:02:06.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:06 smithi156 ceph-mon[32025]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:06.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:06 smithi192 ceph-mon[36545]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:06.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:06 smithi186 ceph-mon[36412]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-05-26T12:02:07.688 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:07 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-26T12:02:07.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:07 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-26T12:02:07.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:07 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]: dispatch 2024-05-26T12:02:08.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.138+0000 7fc70b0e1700 1 -- 172.21.15.156:0/3437832209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70406b220 msgr2=0x7fc70406b6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:08.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.138+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/3437832209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70406b220 0x7fc70406b6a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec009990 tx=0x7fc6ec009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 -- 172.21.15.156:0/3437832209 shutdown_connections 2024-05-26T12:02:08.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/3437832209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc7041103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/3437832209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7041095b0 0x7fc704109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/3437832209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc70406b220 0x7fc70406b6a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 -- 172.21.15.156:0/3437832209 >> 172.21.15.156:0/3437832209 conn(0x7fc7040793f0 msgr2=0x7fc704079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:08.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.139+0000 7fc70b0e1700 1 -- 172.21.15.156:0/3437832209 shutdown_connections 2024-05-26T12:02:08.140 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.140+0000 7fc70b0e1700 1 -- 172.21.15.156:0/3437832209 wait complete. 2024-05-26T12:02:08.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.140+0000 7fc70b0e1700 1 Processor -- start 2024-05-26T12:02:08.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 -- start start 2024-05-26T12:02:08.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc70406b220 0x7fc704117a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:08.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:08.144 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:08.144 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:08.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7041095b0 0x7fc704112bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:08.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc704111580 con 0x7fc70406bbe0 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc704111400 con 0x7fc70406b220 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.141+0000 7fc70b0e1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc704111700 con 0x7fc7041095b0 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42648/0 (socket says 172.21.15.156:42648) 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 -- 172.21.15.156:0/1432568823 learned_addr learned my addr 172.21.15.156:0/1432568823 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc70967e700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7041095b0 0x7fc704112bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 -- 172.21.15.156:0/1432568823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7041095b0 msgr2=0x7fc704112bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:08.145 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7041095b0 0x7fc704112bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 -- 172.21.15.156:0/1432568823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc70406b220 msgr2=0x7fc704117a60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc70406b220 0x7fc704117a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.142+0000 7fc703fff700 1 -- 172.21.15.156:0/1432568823 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc704113430 con 0x7fc70406bbe0 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.143+0000 7fc703fff700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec000c00 tx=0x7fc6ec0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.143+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec027070 con 0x7fc70406bbe0 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.143+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc6ec022470 con 0x7fc70406bbe0 2024-05-26T12:02:08.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.143+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6ec017630 con 0x7fc70406bbe0 2024-05-26T12:02:08.148 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:08.148 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:08.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.148+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7041136c0 con 0x7fc70406bbe0 2024-05-26T12:02:08.149 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.148+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7041b5560 con 0x7fc70406bbe0 2024-05-26T12:02:08.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.153+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fc6ec009d10 con 0x7fc70406bbe0 2024-05-26T12:02:08.154 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.153+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc70410ba90 con 0x7fc70406bbe0 2024-05-26T12:02:08.155 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.155+0000 7fc701ffb700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 0x7fc6f00b4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:08.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.155+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 8277+0+0 (secure 0 0 0) 0x7fc6ec0e2cf0 con 0x7fc70406bbe0 2024-05-26T12:02:08.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.155+0000 7fc708e7d700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 0x7fc6f00b4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:08.156 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.156+0000 7fc708e7d700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 0x7fc6f00b4fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009a70 tx=0x7fc6f4009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:08.165 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.164+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc6ec013070 con 0x7fc70406bbe0 2024-05-26T12:02:08.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.391+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fc70406a260 con 0x7fc70406bbe0 2024-05-26T12:02:08.392 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.392+0000 7fc701ffb700 1 -- 172.21.15.156:0/1432568823 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc70406a260 con 0x7fc70406bbe0 2024-05-26T12:02:08.393 INFO:teuthology.orchestra.run.smithi156.stdout:257698037806 2024-05-26T12:02:08.394 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 msgr2=0x7fc6f00b4fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 0x7fc6f00b4fa0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc6f4009a70 tx=0x7fc6f4009260 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 msgr2=0x7fc704117fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc6ec000c00 tx=0x7fc6ec0057c0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 shutdown_connections 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc6f00b2ae0 0x7fc6f00b4fa0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc7041095b0 0x7fc704112bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc70406b220 0x7fc704117a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 --2- 172.21.15.156:0/1432568823 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc70406bbe0 0x7fc704117fa0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 >> 172.21.15.156:0/1432568823 conn(0x7fc7040793f0 msgr2=0x7fc704101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 shutdown_connections 2024-05-26T12:02:08.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:08.394+0000 7fc70b0e1700 1 -- 172.21.15.156:0/1432568823 wait complete. 2024-05-26T12:02:08.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:08 smithi156 ceph-mon[32025]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:02:08.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:08 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-26T12:02:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:08 smithi156 ceph-mon[32025]: osdmap e82: 12 total, 12 up, 12 in 2024-05-26T12:02:08.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:08 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1432568823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-26T12:02:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:08 smithi192 ceph-mon[36545]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:02:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:08 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-26T12:02:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:08 smithi192 ceph-mon[36545]: osdmap e82: 12 total, 12 up, 12 in 2024-05-26T12:02:08.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:08 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1432568823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-26T12:02:08.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:08 smithi186 ceph-mon[36412]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-05-26T12:02:08.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:08 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.7", "id": [3, 9]}]': finished 2024-05-26T12:02:08.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:08 smithi186 ceph-mon[36412]: osdmap e82: 12 total, 12 up, 12 in 2024-05-26T12:02:08.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:08 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1432568823' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-26T12:02:09.146 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037803 got 257698037806 for osd.9 2024-05-26T12:02:09.146 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:09.658 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:09 smithi156 ceph-mon[32025]: osdmap e83: 12 total, 12 up, 12 in 2024-05-26T12:02:09.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:09 smithi192 ceph-mon[36545]: osdmap e83: 12 total, 12 up, 12 in 2024-05-26T12:02:09.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:09 smithi186 ceph-mon[36412]: osdmap e83: 12 total, 12 up, 12 in 2024-05-26T12:02:10.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 -- 172.21.15.156:0/3246934209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 msgr2=0x7f61e0108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:10.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 --2- 172.21.15.156:0/3246934209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0108a00 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00b0a0 tx=0x7f61dc00b3b0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.253 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 -- 172.21.15.156:0/3246934209 shutdown_connections 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 --2- 172.21.15.156:0/3246934209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e0110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 --2- 172.21.15.156:0/3246934209 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0108a00 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 --2- 172.21.15.156:0/3246934209 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e0101f00 0x7f61e0102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 -- 172.21.15.156:0/3246934209 >> 172.21.15.156:0/3246934209 conn(0x7f61e0077780 msgr2=0x7f61e0077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 -- 172.21.15.156:0/3246934209 shutdown_connections 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.251+0000 7f61e84c1700 1 -- 172.21.15.156:0/3246934209 wait complete. 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 Processor -- start 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 -- start start 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e0101f00 0x7f61e019f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:10.254 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f61e01137b0 con 0x7f61e0108620 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f61e0113630 con 0x7f61e01028c0 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e84c1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f61e0113930 con 0x7f61e0101f00 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:42660/0 (socket says 172.21.15.156:42660) 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 -- 172.21.15.156:0/136839980 learned_addr learned my addr 172.21.15.156:0/136839980 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 -- 172.21.15.156:0/136839980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e0101f00 msgr2=0x7f61e019f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e5a5c700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e0101f00 0x7f61e019f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 -- 172.21.15.156:0/136839980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 msgr2=0x7f61e019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 -- 172.21.15.156:0/136839980 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61e019a210 con 0x7f61e0108620 2024-05-26T12:02:10.255 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e5a5c700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.252+0000 7f61e6a5e700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f61d800c970 tx=0x7f61d800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.253+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8007b00 con 0x7f61e0108620 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.253+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f61d8004ce0 con 0x7f61e0108620 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.253+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61e019a500 con 0x7f61e0108620 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.253+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61d8005530 con 0x7f61e0108620 2024-05-26T12:02:10.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.253+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61e01a6770 con 0x7f61e0108620 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.254+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61e006a230 con 0x7f61e0108620 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.255+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f61d8004750 con 0x7f61e0108620 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.256+0000 7f61d77fe700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 0x7f61cc0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.256+0000 7f61e625d700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 0x7f61cc0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.256+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f61d80e2420 con 0x7f61e0108620 2024-05-26T12:02:10.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.257+0000 7f61e625d700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 0x7f61cc0b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00b070 tx=0x7f61dc0060c0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:10.260 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.260+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f61d8014030 con 0x7f61e0108620 2024-05-26T12:02:10.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.470+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f61e002e560 con 0x7f61e0108620 2024-05-26T12:02:10.471 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.471+0000 7f61d77fe700 1 -- 172.21.15.156:0/136839980 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f61d80a9fb0 con 0x7f61e0108620 2024-05-26T12:02:10.472 INFO:teuthology.orchestra.run.smithi156.stdout:137438953544 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 msgr2=0x7f61cc0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 0x7f61cc0b4dd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f61dc00b070 tx=0x7f61dc0060c0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 msgr2=0x7f61e0199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f61d800c970 tx=0x7f61d800cd30 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 shutdown_connections 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f61cc0b2910 0x7f61cc0b4dd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f61e0108620 0x7f61e0199ab0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f61e0101f00 0x7f61e019f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 --2- 172.21.15.156:0/136839980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e01028c0 0x7f61e019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:10.474 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 >> 172.21.15.156:0/136839980 conn(0x7f61e0077780 msgr2=0x7f61e0100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:10.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 shutdown_connections 2024-05-26T12:02:10.475 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:10.473+0000 7f61e84c1700 1 -- 172.21.15.156:0/136839980 wait complete. 2024-05-26T12:02:10.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:10 smithi156 ceph-mon[32025]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-26T12:02:10.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:10 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/136839980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-26T12:02:10.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:10 smithi192 ceph-mon[36545]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-26T12:02:10.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:10 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/136839980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-26T12:02:10.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:10 smithi186 ceph-mon[36412]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-05-26T12:02:10.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:10 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/136839980' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-26T12:02:11.362 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953539 got 137438953544 for osd.4 2024-05-26T12:02:11.362 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:12.430 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.430+0000 7f0625f5b700 1 -- 172.21.15.156:0/3845375182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 msgr2=0x7f06201088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.430+0000 7f0625f5b700 1 --2- 172.21.15.156:0/3845375182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201088a0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0610009990 tx=0x7f0610009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 -- 172.21.15.156:0/3845375182 shutdown_connections 2024-05-26T12:02:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 --2- 172.21.15.156:0/3845375182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062010f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 --2- 172.21.15.156:0/3845375182 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201088a0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 --2- 172.21.15.156:0/3845375182 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 0x7f06201030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 -- 172.21.15.156:0/3845375182 >> 172.21.15.156:0/3845375182 conn(0x7f0620078ab0 msgr2=0x7f06200ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:12.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 -- 172.21.15.156:0/3845375182 shutdown_connections 2024-05-26T12:02:12.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 -- 172.21.15.156:0/3845375182 wait complete. 2024-05-26T12:02:12.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.431+0000 7f0625f5b700 1 Processor -- start 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 -- start start 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 0x7f062019abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f06201126a0 con 0x7f06201084c0 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f0620112520 con 0x7f0620103600 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f0625f5b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f0620112820 con 0x7f0620102c40 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f061effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f061effd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.156:37910/0 (socket says 172.21.15.156:37910) 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f061effd700 1 -- 172.21.15.156:0/191294775 learned_addr learned my addr 172.21.15.156:0/191294775 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:12.434 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f061f7fe700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 0x7f062019abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.432+0000 7f061ffff700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 -- 172.21.15.156:0/191294775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 msgr2=0x7f062019abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 0x7f062019abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 -- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 msgr2=0x7f06201a5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201a5c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 -- 172.21.15.156:0/191294775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06201a6180 con 0x7f0620103600 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061effd700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f061400b530 tx=0x7f061400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614015020 con 0x7f0620103600 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0614011ce0 con 0x7f0620103600 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06201989b0 con 0x7f0620103600 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614010dd0 con 0x7f0620103600 2024-05-26T12:02:12.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.433+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0620198ec0 con 0x7f0620103600 2024-05-26T12:02:12.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.434+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f062010aa10 con 0x7f0620103600 2024-05-26T12:02:12.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.436+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f0614018020 con 0x7f0620103600 2024-05-26T12:02:12.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.437+0000 7f061cff9700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 0x7f06080b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:12.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.437+0000 7f061f7fe700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 0x7f06080b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:12.437 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.437+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f06140e1f80 con 0x7f0620103600 2024-05-26T12:02:12.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.437+0000 7f061f7fe700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 0x7f06080b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f061000b5c0 tx=0x7f0610005fd0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:12.441 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.440+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f061401f020 con 0x7f0620103600 2024-05-26T12:02:12.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.658+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f062010a3f0 con 0x7f0620103600 2024-05-26T12:02:12.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.659+0000 7f061cff9700 1 -- 172.21.15.156:0/191294775 <== mon.2 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f06140a9bc0 con 0x7f0620103600 2024-05-26T12:02:12.660 INFO:teuthology.orchestra.run.smithi156.stdout:231928234036 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 msgr2=0x7f06080b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 0x7f06080b4dd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f061000b5c0 tx=0x7f0610005fd0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 msgr2=0x7f062019b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f061400b530 tx=0x7f061400b840 comp rx=0 tx=0).stop 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 shutdown_connections 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f06080b2910 0x7f06080b4dd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f06201084c0 0x7f06201a5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f0620102c40 0x7f062019abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 --2- 172.21.15.156:0/191294775 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0620103600 0x7f062019b100 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:12.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 >> 172.21.15.156:0/191294775 conn(0x7f0620078ab0 msgr2=0x7f06201002c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:12.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.661+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 shutdown_connections 2024-05-26T12:02:12.663 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:12.662+0000 7f0625f5b700 1 -- 172.21.15.156:0/191294775 wait complete. 2024-05-26T12:02:12.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:12 smithi156 ceph-mon[32025]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-26T12:02:12.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:12 smithi192 ceph-mon[36545]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-26T12:02:12.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:12 smithi186 ceph-mon[36412]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-05-26T12:02:13.746 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:13 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/191294775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-26T12:02:13.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:13 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/191294775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-26T12:02:13.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:13 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/191294775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-26T12:02:14.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:14 smithi192 ceph-mon[36545]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-05-26T12:02:14.861 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:14 smithi156 ceph-mon[32025]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-05-26T12:02:14.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:14 smithi186 ceph-mon[36412]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 9 op/s 2024-05-26T12:02:15.409 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:15.887 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234031 got 231928234036 for osd.8 2024-05-26T12:02:15.887 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:16.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:16 smithi156 ceph-mon[32025]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-05-26T12:02:16.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:16 smithi192 ceph-mon[36545]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-05-26T12:02:16.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:16 smithi186 ceph-mon[36412]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 9 op/s 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 -- 172.21.15.156:0/2950418186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a441035d0 msgr2=0x7f4a44103a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/2950418186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a441035d0 0x7f4a44103a50 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4a380099c0 tx=0x7f4a38009cd0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 -- 172.21.15.156:0/2950418186 shutdown_connections 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/2950418186 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a44103f90 0x7f4a44110440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/2950418186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44109a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/2950418186 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a441035d0 0x7f4a44103a50 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 -- 172.21.15.156:0/2950418186 >> 172.21.15.156:0/2950418186 conn(0x7f4a4407a730 msgr2=0x7f4a4407ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 -- 172.21.15.156:0/2950418186 shutdown_connections 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.536+0000 7f4a48fdc700 1 -- 172.21.15.156:0/2950418186 wait complete. 2024-05-26T12:02:17.659 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 Processor -- start 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 -- start start 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a441035d0 0x7f4a4419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44113790 con 0x7f4a44103f90 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44113610 con 0x7f4a44109670 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a48fdc700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f4a44113910 con 0x7f4a441035d0 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a41d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a41d9b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41288/0 (socket says 172.21.15.156:41288) 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a41d9b700 1 -- 172.21.15.156:0/1569718056 learned_addr learned my addr 172.21.15.156:0/1569718056 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a4259c700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a441035d0 0x7f4a4419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.537+0000 7f4a42d9d700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.660 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 -- 172.21.15.156:0/1569718056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a441035d0 msgr2=0x7f4a4419f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a441035d0 0x7f4a4419f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 -- 172.21.15.156:0/1569718056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 msgr2=0x7f4a44199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 -- 172.21.15.156:0/1569718056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a4419a1b0 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a42d9d700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a41d9b700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800b580 tx=0x7f4a380047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a38027070 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a4419a440 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a380040a0 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a3800f460 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.538+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a441a6760 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.539+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a4406a230 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.540+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f4a3802a070 con 0x7f4a44103f90 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.541+0000 7f4a337fe700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 0x7f4a2c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.661 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.541+0000 7f4a4259c700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 0x7f4a2c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.541+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f4a380e3880 con 0x7f4a44103f90 2024-05-26T12:02:17.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.542+0000 7f4a4259c700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 0x7f4a2c0b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005950 tx=0x7f4a3400a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:17.662 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.545+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4a38031070 con 0x7f4a44103f90 2024-05-26T12:02:17.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.758+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f4a4402e560 con 0x7f4a44103f90 2024-05-26T12:02:17.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.758+0000 7f4a337fe700 1 -- 172.21.15.156:0/1569718056 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f4a38013070 con 0x7f4a44103f90 2024-05-26T12:02:17.759 INFO:teuthology.orchestra.run.smithi156.stdout:279172874282 2024-05-26T12:02:17.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.760+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 msgr2=0x7f4a2c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.760+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 0x7f4a2c0b4dd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4a34005950 tx=0x7f4a3400a300 comp rx=0 tx=0).stop 2024-05-26T12:02:17.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.760+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 msgr2=0x7f4a4419fdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.760+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4a3800b580 tx=0x7f4a380047b0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 shutdown_connections 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f4a2c0b2910 0x7f4a2c0b4dd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a44109670 0x7f4a44199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4a441035d0 0x7f4a4419f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 --2- 172.21.15.156:0/1569718056 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4a44103f90 0x7f4a4419fdd0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 >> 172.21.15.156:0/1569718056 conn(0x7f4a4407a730 msgr2=0x7f4a44102370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 shutdown_connections 2024-05-26T12:02:17.762 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.761+0000 7f4a48fdc700 1 -- 172.21.15.156:0/1569718056 wait complete. 2024-05-26T12:02:17.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/2709107951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84103600 msgr2=0x7f7d8410f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.852 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/2709107951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84103600 0x7f7d8410f2e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c00b0a0 tx=0x7f7d7c00b3b0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/2709107951 shutdown_connections 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/2709107951 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84103600 0x7f7d8410f2e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/2709107951 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/2709107951 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84102c40 0x7f7d841030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/2709107951 >> 172.21.15.156:0/2709107951 conn(0x7f7d84078ab0 msgr2=0x7f7d840ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/2709107951 shutdown_connections 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.851+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/2709107951 wait complete. 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 Processor -- start 2024-05-26T12:02:17.853 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 -- start start 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84103600 0x7f7d8419fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f7d84112650 con 0x7f7d84102c40 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841124d0 con 0x7f7d841084c0 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d8a9f3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f7d841127d0 con 0x7f7d84103600 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d83fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d83fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41304/0 (socket says 172.21.15.156:41304) 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.852+0000 7f7d83fff700 1 -- 172.21.15.156:0/1380508480 learned_addr learned my addr 172.21.15.156:0/1380508480 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:17.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d88f90700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d837fe700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84103600 0x7f7d8419fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 -- 172.21.15.156:0/1380508480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84103600 msgr2=0x7f7d8419fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84103600 0x7f7d8419fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 -- 172.21.15.156:0/1380508480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 msgr2=0x7f7d841999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 -- 172.21.15.156:0/1380508480 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8419a150 con 0x7f7d84102c40 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d88f90700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d83fff700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400d980 tx=0x7f7d7400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74017030 con 0x7f7d84102c40 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7d740044d0 con 0x7f7d84102c40 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d74003aa0 con 0x7f7d84102c40 2024-05-26T12:02:17.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d8419a440 con 0x7f7d84102c40 2024-05-26T12:02:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.853+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d841a66b0 con 0x7f7d84102c40 2024-05-26T12:02:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.854+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d84103c50 con 0x7f7d84102c40 2024-05-26T12:02:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.855+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f7d7401a070 con 0x7f7d84102c40 2024-05-26T12:02:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.856+0000 7f7d817fa700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 0x7f7d6c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:17.857 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.856+0000 7f7d837fe700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 0x7f7d6c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:17.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.856+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f7d740e21b0 con 0x7f7d84102c40 2024-05-26T12:02:17.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.857+0000 7f7d837fe700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 0x7f7d6c0b4dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006fd0 tx=0x7f7d78008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:17.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:17.859+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f7d740a9d40 con 0x7f7d84102c40 2024-05-26T12:02:18.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.070+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f7d8406a230 con 0x7f7d84102c40 2024-05-26T12:02:18.071 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.070+0000 7f7d817fa700 1 -- 172.21.15.156:0/1380508480 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f7d740a9640 con 0x7f7d84102c40 2024-05-26T12:02:18.071 INFO:teuthology.orchestra.run.smithi156.stdout:304942678053 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 msgr2=0x7f7d6c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 0x7f7d6c0b4dd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7d78006fd0 tx=0x7f7d78008040 comp rx=0 tx=0).stop 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 msgr2=0x7f7d8419f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7d7400d980 tx=0x7f7d7400dd40 comp rx=0 tx=0).stop 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 shutdown_connections 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f7d6c0b2910 0x7f7d6c0b4dd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:18.074 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d841084c0 0x7f7d841999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:18.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f7d84102c40 0x7f7d8419f7e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:18.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 --2- 172.21.15.156:0/1380508480 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f7d84103600 0x7f7d8419fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:18.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 >> 172.21.15.156:0/1380508480 conn(0x7f7d84078ab0 msgr2=0x7f7d84100800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:18.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 shutdown_connections 2024-05-26T12:02:18.075 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:18.073+0000 7f7d8a9f3700 1 -- 172.21.15.156:0/1380508480 wait complete. 2024-05-26T12:02:18.535 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874276 got 279172874282 for osd.10 2024-05-26T12:02:18.535 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:18.745 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:18.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:18 smithi156 ceph-mon[32025]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-26T12:02:18.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:18 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1569718056' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-26T12:02:18.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:18 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1380508480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-26T12:02:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:18 smithi192 ceph-mon[36545]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-26T12:02:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:18 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1569718056' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-26T12:02:18.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:18 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1380508480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-26T12:02:18.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:18 smithi186 ceph-mon[36412]: pgmap v348: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-05-26T12:02:18.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:18 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1569718056' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-26T12:02:18.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:18 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1380508480' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-26T12:02:20.622 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:20 smithi156 ceph-mon[32025]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-26T12:02:20.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:20 smithi192 ceph-mon[36545]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-26T12:02:20.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:20 smithi186 ceph-mon[36412]: pgmap v349: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-05-26T12:02:21.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 -- 172.21.15.156:0/3064445435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c103010 msgr2=0x7f032c103490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:21.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 --2- 172.21.15.156:0/3064445435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c103010 0x7f032c103490 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f03180099c0 tx=0x7f0318009cd0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.921 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 -- 172.21.15.156:0/3064445435 shutdown_connections 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 --2- 172.21.15.156:0/3064445435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c1039d0 0x7f032c10f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 --2- 172.21.15.156:0/3064445435 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c108730 0x7f032c108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 --2- 172.21.15.156:0/3064445435 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c103010 0x7f032c103490 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 -- 172.21.15.156:0/3064445435 >> 172.21.15.156:0/3064445435 conn(0x7f032c077780 msgr2=0x7f032c077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 -- 172.21.15.156:0/3064445435 shutdown_connections 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 -- 172.21.15.156:0/3064445435 wait complete. 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.919+0000 7f0332b95700 1 Processor -- start 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 -- start start 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c103010 0x7f032c104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c1039d0 0x7f032c1046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:21.922 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f032c1127b0 con 0x7f032c108730 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f032c112630 con 0x7f032c103010 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0332b95700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f032c112930 con 0x7f032c1039d0 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41324/0 (socket says 172.21.15.156:41324) 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 -- 172.21.15.156:0/2089579969 learned_addr learned my addr 172.21.15.156:0/2089579969 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0330931700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c103010 0x7f032c104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f032bfff700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c1039d0 0x7f032c1046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 -- 172.21.15.156:0/2089579969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c1039d0 msgr2=0x7f032c1046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c1039d0 0x7f032c1046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 -- 172.21.15.156:0/2089579969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c103010 msgr2=0x7f032c104190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c103010 0x7f032c104190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 -- 172.21.15.156:0/2089579969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f032c1a81a0 con 0x7f032c108730 2024-05-26T12:02:21.923 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.920+0000 7f0331132700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0324007d60 tx=0x7f032400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:21.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.921+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f032400ac90 con 0x7f032c108730 2024-05-26T12:02:21.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.921+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f032400ae30 con 0x7f032c108730 2024-05-26T12:02:21.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.921+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f032c1a8490 con 0x7f032c108730 2024-05-26T12:02:21.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.921+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0324012420 con 0x7f032c108730 2024-05-26T12:02:21.924 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.921+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f032c1a89a0 con 0x7f032c108730 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.922+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f032c06a230 con 0x7f032c108730 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.923+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f0324003a00 con 0x7f032c108730 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.924+0000 7f0329ffb700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 0x7f031c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.924+0000 7f0330931700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 0x7f031c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.924+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f03240e2450 con 0x7f032c108730 2024-05-26T12:02:21.925 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.924+0000 7f0330931700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 0x7f031c0b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0320005950 tx=0x7f032000a300 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:21.929 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:21.928+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f03240aa0f0 con 0x7f032c108730 2024-05-26T12:02:22.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.143+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f032c1056c0 con 0x7f032c108730 2024-05-26T12:02:22.144 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.143+0000 7f0329ffb700 1 -- 172.21.15.156:0/2089579969 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f03240a99f0 con 0x7f032c108730 2024-05-26T12:02:22.144 INFO:teuthology.orchestra.run.smithi156.stdout:55834574938 2024-05-26T12:02:22.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 msgr2=0x7f031c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 0x7f031c0b4dd0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0320005950 tx=0x7f032000a300 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 msgr2=0x7f032c104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0324007d60 tx=0x7f032400a340 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 shutdown_connections 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f031c0b2910 0x7f031c0b4dd0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f032c108730 0x7f032c104c10 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f032c103010 0x7f032c104190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 --2- 172.21.15.156:0/2089579969 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f032c1039d0 0x7f032c1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 >> 172.21.15.156:0/2089579969 conn(0x7f032c077780 msgr2=0x7f032c102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 shutdown_connections 2024-05-26T12:02:22.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.146+0000 7f0332b95700 1 -- 172.21.15.156:0/2089579969 wait complete. 2024-05-26T12:02:22.159 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678050 got 304942678053 for osd.11 2024-05-26T12:02:22.159 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:22.776 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:22 smithi156 ceph-mon[32025]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:22.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:22 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2089579969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-26T12:02:22.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574932 got 55834574938 for osd.1 2024-05-26T12:02:22.803 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:22 smithi192 ceph-mon[36545]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:22 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2089579969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-26T12:02:22.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:22 smithi186 ceph-mon[36412]: pgmap v350: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:22.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:22 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2089579969' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-26T12:02:23.304 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:23.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 -- 172.21.15.156:0/4246887609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 msgr2=0x7fc74810f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:23.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 --2- 172.21.15.156:0/4246887609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74810f2f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fc74000b0a0 tx=0x7fc74000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 -- 172.21.15.156:0/4246887609 shutdown_connections 2024-05-26T12:02:23.818 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 --2- 172.21.15.156:0/4246887609 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74810f2f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 --2- 172.21.15.156:0/4246887609 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 0x7fc748108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 --2- 172.21.15.156:0/4246887609 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 0x7fc7481028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 -- 172.21.15.156:0/4246887609 >> 172.21.15.156:0/4246887609 conn(0x7fc7480793f0 msgr2=0x7fc748079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 -- 172.21.15.156:0/4246887609 shutdown_connections 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.988+0000 7fc750553700 1 -- 172.21.15.156:0/4246887609 wait complete. 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 Processor -- start 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 -- start start 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 0x7fc74819f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:23.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 0x7fc7481999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:23.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc7481125c0 con 0x7fc748102e30 2024-05-26T12:02:23.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc748112440 con 0x7fc7481084b0 2024-05-26T12:02:23.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc750553700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc748112740 con 0x7fc748102470 2024-05-26T12:02:23.820 INFO:teuthology.orchestra.run.smithi156.stdout:184683593792 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:41330/0 (socket says 172.21.15.156:41330) 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 -- 172.21.15.156:0/1697498777 learned_addr learned my addr 172.21.15.156:0/1697498777 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74eaf0700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 0x7fc7481999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74e2ef700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 0x7fc74819f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 -- 172.21.15.156:0/1697498777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 msgr2=0x7fc74819f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 0x7fc74819f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 -- 172.21.15.156:0/1697498777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 msgr2=0x7fc7481999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:23.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 0x7fc7481999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.989+0000 7fc74daee700 1 -- 172.21.15.156:0/1697498777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74819a110 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc74daee700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c920 tx=0x7fc73800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738007b00 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc738004ce0 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc74819a400 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc738005650 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.990+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7481a66c0 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.991+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74806a230 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.992+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fc738004750 con 0x7fc748102e30 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.993+0000 7fc73f7fe700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 0x7fc7340b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.993+0000 7fc74e2ef700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 0x7fc7340b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:23.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.993+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fc7380e2420 con 0x7fc748102e30 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.994+0000 7fc74e2ef700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 0x7fc7340b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc744005fd0 tx=0x7fc744005f60 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:22.997+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc738014030 con 0x7fc748102e30 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.209+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc74802e560 con 0x7fc748102e30 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.209+0000 7fc73f7fe700 1 -- 172.21.15.156:0/1697498777 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc7380a9fb0 con 0x7fc748102e30 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 msgr2=0x7fc7340b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 0x7fc7340b4dd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc744005fd0 tx=0x7fc744005f60 comp rx=0 tx=0).stop 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 msgr2=0x7fc74819fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fc73800c920 tx=0x7fc73800cce0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 shutdown_connections 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7340b2910 0x7fc7340b4dd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7481084b0 0x7fc7481999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc748102470 0x7fc74819f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 --2- 172.21.15.156:0/1697498777 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc748102e30 0x7fc74819fd30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:23.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 >> 172.21.15.156:0/1697498777 conn(0x7fc7480793f0 msgr2=0x7fc74806bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:23.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 shutdown_connections 2024-05-26T12:02:23.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:23.212+0000 7fc750553700 1 -- 172.21.15.156:0/1697498777 wait complete. 2024-05-26T12:02:23.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:23 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1697498777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-26T12:02:24.044 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:23 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1697498777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-26T12:02:24.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:23 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1697498777' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-26T12:02:24.727 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:24 smithi156 ceph-mon[32025]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:24.753 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593787 got 184683593792 for osd.6 2024-05-26T12:02:24.753 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:24.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:24 smithi186 ceph-mon[36412]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:25.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:24 smithi192 ceph-mon[36545]: pgmap v351: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 -- 172.21.15.156:0/1095163339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 msgr2=0x7fe4241028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 --2- 172.21.15.156:0/1095163339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe4241028c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe418009990 tx=0x7fe418009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 -- 172.21.15.156:0/1095163339 shutdown_connections 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 --2- 172.21.15.156:0/1095163339 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe424102e00 0x7fe42410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 --2- 172.21.15.156:0/1095163339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424108520 0x7fe424108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 --2- 172.21.15.156:0/1095163339 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe4241028c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 -- 172.21.15.156:0/1095163339 >> 172.21.15.156:0/1095163339 conn(0x7fe4240793f0 msgr2=0x7fe424079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 -- 172.21.15.156:0/1095163339 shutdown_connections 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.231+0000 7fe42a321700 1 -- 172.21.15.156:0/1095163339 wait complete. 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 Processor -- start 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 -- start start 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 0x7fe42419fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe424108520 0x7fe424199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fe4241125d0 con 0x7fe424102440 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fe424112450 con 0x7fe424102e00 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe42a321700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fe424112750 con 0x7fe424108520 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe423fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe423fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55630/0 (socket says 172.21.15.156:55630) 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.232+0000 7fe423fff700 1 -- 172.21.15.156:0/188230102 learned_addr learned my addr 172.21.15.156:0/188230102 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:25.273 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe4237fe700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 0x7fe42419fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 -- 172.21.15.156:0/188230102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe424108520 msgr2=0x7fe424199ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe424108520 0x7fe424199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 -- 172.21.15.156:0/188230102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 msgr2=0x7fe42419fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 0x7fe42419fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 -- 172.21.15.156:0/188230102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe42419a220 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe4237fe700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 0x7fe42419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe423fff700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe41400c9b0 tx=0x7fe41400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414007b00 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe42419a510 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe414007ca0 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe414018730 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.233+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4241a67d0 con 0x7fe424102440 2024-05-26T12:02:25.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.234+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe42406a230 con 0x7fe424102440 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.236+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fe41400f450 con 0x7fe424102440 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.237+0000 7fe4217fa700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 0x7fe40c0b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.237+0000 7fe4237fe700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 0x7fe40c0b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.237+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fe4140e26f0 con 0x7fe424102440 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.237+0000 7fe4237fe700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 0x7fe40c0b4e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe418000c00 tx=0x7fe418005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:25.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.240+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fe414014030 con 0x7fe424102440 2024-05-26T12:02:25.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.458+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe42419aff0 con 0x7fe424102440 2024-05-26T12:02:25.459 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.458+0000 7fe4217fa700 1 -- 172.21.15.156:0/188230102 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe4140aa2a0 con 0x7fe424102440 2024-05-26T12:02:25.459 INFO:teuthology.orchestra.run.smithi156.stdout:206158430268 2024-05-26T12:02:25.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 msgr2=0x7fe40c0b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:25.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 0x7fe40c0b4e50 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe418000c00 tx=0x7fe418005fb0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 msgr2=0x7fe42419f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:25.461 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe41400c9b0 tx=0x7fe41400cd70 comp rx=0 tx=0).stop 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 shutdown_connections 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fe40c0b2990 0x7fe40c0b4e50 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fe424108520 0x7fe424199ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fe424102440 0x7fe42419f900 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 --2- 172.21.15.156:0/188230102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe424102e00 0x7fe42419fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 >> 172.21.15.156:0/188230102 conn(0x7fe4240793f0 msgr2=0x7fe42406b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 shutdown_connections 2024-05-26T12:02:25.462 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:25.461+0000 7fe42a321700 1 -- 172.21.15.156:0/188230102 wait complete. 2024-05-26T12:02:25.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:25 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/188230102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-26T12:02:25.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:25 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/188230102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-26T12:02:26.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:25 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/188230102' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-26T12:02:26.225 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430262 got 206158430268 for osd.7 2024-05-26T12:02:26.226 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:26.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- 172.21.15.156:0/3792172304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804109520 msgr2=0x7f4804109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 --2- 172.21.15.156:0/3792172304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804109520 0x7f4804109900 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f480000b0d0 tx=0x7f480000b3e0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- 172.21.15.156:0/3792172304 shutdown_connections 2024-05-26T12:02:26.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 --2- 172.21.15.156:0/3792172304 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f48041024f0 0x7f48041103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.491 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 --2- 172.21.15.156:0/3792172304 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804109520 0x7f4804109900 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 --2- 172.21.15.156:0/3792172304 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4804101b30 0x7f4804101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- 172.21.15.156:0/3792172304 >> 172.21.15.156:0/3792172304 conn(0x7f4804078ab0 msgr2=0x7f48040ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- 172.21.15.156:0/3792172304 shutdown_connections 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- 172.21.15.156:0/3792172304 wait complete. 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 Processor -- start 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.462+0000 7f480b2fd700 1 -- start start 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 0x7f480419fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4804109520 0x7f4804199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f4804113760 con 0x7f4804101b30 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f48041135e0 con 0x7f48041024f0 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f48041138e0 con 0x7f4804109520 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.492 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55644/0 (socket says 172.21.15.156:55644) 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 -- 172.21.15.156:0/534555191 learned_addr learned my addr 172.21.15.156:0/534555191 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4808898700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 0x7f480419fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 -- 172.21.15.156:0/534555191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4804109520 msgr2=0x7f4804199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4804109520 0x7f4804199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 -- 172.21.15.156:0/534555191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 msgr2=0x7f480419fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 0x7f480419fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 -- 172.21.15.156:0/534555191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480419a1e0 con 0x7f4804101b30 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4808898700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 0x7f480419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f4809099700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f480000d010 tx=0x7f4800007ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480001f070 con 0x7f4804101b30 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4800003c00 con 0x7f4804101b30 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480419a470 con 0x7f4804101b30 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480001b6a0 con 0x7f4804101b30 2024-05-26T12:02:26.493 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.463+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48041a6790 con 0x7f4804101b30 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.464+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4804104e50 con 0x7f4804101b30 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.466+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f4800022070 con 0x7f4804101b30 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.467+0000 7f47fa7fc700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 0x7f47f00b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.467+0000 7f4808898700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 0x7f47f00b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.467+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f480000f020 con 0x7f4804101b30 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.467+0000 7f4808898700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 0x7f47f00b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005950 tx=0x7f47f40058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:26.494 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.471+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f4800029020 con 0x7f4804101b30 2024-05-26T12:02:26.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.690+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f480419afb0 con 0x7f4804101b30 2024-05-26T12:02:26.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.690+0000 7f47fa7fc700 1 -- 172.21.15.156:0/534555191 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f48000cd090 con 0x7f4804101b30 2024-05-26T12:02:26.691 INFO:teuthology.orchestra.run.smithi156.stdout:158913790022 2024-05-26T12:02:26.693 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 msgr2=0x7f47f00b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 0x7f47f00b4dd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f47f4005950 tx=0x7f47f40058e0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 msgr2=0x7f480419f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f480000d010 tx=0x7f4800007ad0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 shutdown_connections 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f47f00b2910 0x7f47f00b4dd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f4804109520 0x7f4804199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f4804101b30 0x7f480419f8c0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 --2- 172.21.15.156:0/534555191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48041024f0 0x7f480419fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 >> 172.21.15.156:0/534555191 conn(0x7f4804078ab0 msgr2=0x7f4804100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 shutdown_connections 2024-05-26T12:02:26.694 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.692+0000 7f480b2fd700 1 -- 172.21.15.156:0/534555191 wait complete. 2024-05-26T12:02:26.708 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:26 smithi156 ceph-mon[32025]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:26.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 -- 172.21.15.156:0/104130156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102440 msgr2=0x7f929c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.932 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 --2- 172.21.15.156:0/104130156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102440 0x7f929c1028c0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f9288009990 tx=0x7f9288009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 -- 172.21.15.156:0/104130156 shutdown_connections 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 --2- 172.21.15.156:0/104130156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102e00 0x7f929c10f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 --2- 172.21.15.156:0/104130156 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 0x7f929c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 --2- 172.21.15.156:0/104130156 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102440 0x7f929c1028c0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 -- 172.21.15.156:0/104130156 >> 172.21.15.156:0/104130156 conn(0x7f929c0793f0 msgr2=0x7f929c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 -- 172.21.15.156:0/104130156 shutdown_connections 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.931+0000 7f92a3034700 1 -- 172.21.15.156:0/104130156 wait complete. 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 Processor -- start 2024-05-26T12:02:26.933 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 -- start start 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102440 0x7f929c19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 0x7f929c198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f929c1126d0 con 0x7f929c102e00 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112550 con 0x7f929c102440 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f92a3034700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f929c112850 con 0x7f929c108520 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f929bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.935 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f929bfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55662/0 (socket says 172.21.15.156:55662) 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.932+0000 7f929bfff700 1 -- 172.21.15.156:0/540585822 learned_addr learned my addr 172.21.15.156:0/540585822 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f92a0dd0700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102440 0x7f929c19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f92a15d1700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 0x7f929c198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 -- 172.21.15.156:0/540585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 msgr2=0x7f929c198a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 0x7f929c198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 -- 172.21.15.156:0/540585822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102440 msgr2=0x7f929c19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102440 0x7f929c19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 -- 172.21.15.156:0/540585822 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f929c1991f0 con 0x7f929c102e00 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f929bfff700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9288000c00 tx=0x7f92880047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9288027070 con 0x7f929c102e00 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92880041d0 con 0x7f929c102e00 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f928800f460 con 0x7f929c102e00 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f929c199480 con 0x7f929c102e00 2024-05-26T12:02:26.936 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.933+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f929c06b220 con 0x7f929c102e00 2024-05-26T12:02:26.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.934+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f929c103b60 con 0x7f929c102e00 2024-05-26T12:02:26.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.936+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f928802a070 con 0x7f929c102e00 2024-05-26T12:02:26.937 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.936+0000 7f9299ffb700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 0x7f928c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:26.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.936+0000 7f92a0dd0700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 0x7f928c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:26.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.936+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f92880e3810 con 0x7f929c102e00 2024-05-26T12:02:26.938 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.937+0000 7f92a0dd0700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 0x7f928c0b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9290006fd0 tx=0x7f9290008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:26.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:26.940+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f9288031070 con 0x7f929c102e00 2024-05-26T12:02:26.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:26 smithi186 ceph-mon[36412]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:27.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:26 smithi192 ceph-mon[36545]: pgmap v352: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:27.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.149+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f929c06a230 con 0x7f929c102e00 2024-05-26T12:02:27.150 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.149+0000 7f9299ffb700 1 -- 172.21.15.156:0/540585822 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9288013070 con 0x7f929c102e00 2024-05-26T12:02:27.150 INFO:teuthology.orchestra.run.smithi156.stdout:77309411414 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 msgr2=0x7f928c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 0x7f928c0b4dd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f9290006fd0 tx=0x7f9290008040 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 msgr2=0x7f929c19ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f9288000c00 tx=0x7f92880047b0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 shutdown_connections 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f928c0b2910 0x7f928c0b4dd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f929c108520 0x7f929c198a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f929c102440 0x7f929c19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 --2- 172.21.15.156:0/540585822 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f929c102e00 0x7f929c19ee10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 >> 172.21.15.156:0/540585822 conn(0x7f929c0793f0 msgr2=0x7f929c100650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 shutdown_connections 2024-05-26T12:02:27.153 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.152+0000 7f92a3034700 1 -- 172.21.15.156:0/540585822 wait complete. 2024-05-26T12:02:27.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.578+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2319500382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1084d0 msgr2=0x7f3dbc1088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.578+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2319500382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc1088b0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009990 tx=0x7f3dac009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.579 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.578+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2319500382 shutdown_connections 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.578+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2319500382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc102e90 0x7f3dbc10f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2319500382 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc1088b0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2319500382 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2319500382 >> 172.21.15.156:0/2319500382 conn(0x7f3dbc078490 msgr2=0x7f3dbc0788a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2319500382 shutdown_connections 2024-05-26T12:02:27.580 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2319500382 wait complete. 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 Processor -- start 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.579+0000 7f3dc20eb700 1 -- start start 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc102e90 0x7f3dbc19fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc112660 con 0x7f3dbc1024d0 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1124e0 con 0x7f3dbc1084d0 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1127e0 con 0x7f3dbc102e90 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55676/0 (socket says 172.21.15.156:55676) 2024-05-26T12:02:27.582 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 -- 172.21.15.156:0/2064595086 learned_addr learned my addr 172.21.15.156:0/2064595086 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbaffd700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc102e90 0x7f3dbc19fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbbfff700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 -- 172.21.15.156:0/2064595086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc102e90 msgr2=0x7f3dbc19fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc102e90 0x7f3dbc19fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 -- 172.21.15.156:0/2064595086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc1084d0 msgr2=0x7f3dbc199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 -- 172.21.15.156:0/2064595086 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc19a1e0 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dbb7fe700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3dac000c00 tx=0x7f3dac0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac027070 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3dac003f20 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac022cf0 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.580+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc19a470 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.581+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc1a6790 con 0x7f3dbc1024d0 2024-05-26T12:02:27.583 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.581+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dbc06a230 con 0x7f3dbc1024d0 2024-05-26T12:02:27.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.583+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7f3dac02a070 con 0x7f3dbc1024d0 2024-05-26T12:02:27.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.584+0000 7f3db8ff9700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 0x7f3da40b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:27.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.584+0000 7f3dbaffd700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 0x7f3da40b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:27.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.584+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f3dac0e3540 con 0x7f3dbc1024d0 2024-05-26T12:02:27.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.584+0000 7f3dbaffd700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 0x7f3da40b4e50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc078be0 tx=0x7f3db0009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:27.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.588+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f3dac013070 con 0x7f3dbc1024d0 2024-05-26T12:02:27.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:27 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/534555191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-26T12:02:27.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:27 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/540585822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-26T12:02:27.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.795+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f3dbc02e560 con 0x7f3dbc1024d0 2024-05-26T12:02:27.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.795+0000 7f3db8ff9700 1 -- 172.21.15.156:0/2064595086 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f3dac0ab100 con 0x7f3dbc1024d0 2024-05-26T12:02:27.796 INFO:teuthology.orchestra.run.smithi156.stdout:34359738463 2024-05-26T12:02:27.798 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 msgr2=0x7f3da40b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 0x7f3da40b4e50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc078be0 tx=0x7f3db0009190 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 msgr2=0x7f3dbc19f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3dac000c00 tx=0x7f3dac0047b0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 shutdown_connections 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f3da40b2990 0x7f3da40b4e50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dbc1084d0 0x7f3dbc199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f3dbc1024d0 0x7f3dbc19f8c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 --2- 172.21.15.156:0/2064595086 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f3dbc102e90 0x7f3dbc19fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 >> 172.21.15.156:0/2064595086 conn(0x7f3dbc078490 msgr2=0x7f3dbc06bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 shutdown_connections 2024-05-26T12:02:27.799 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:27.798+0000 7f3dc20eb700 1 -- 172.21.15.156:0/2064595086 wait complete. 2024-05-26T12:02:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:27 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/534555191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-26T12:02:27.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:27 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/540585822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-26T12:02:28.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:27 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/534555191' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-26T12:02:28.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:27 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/540585822' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-26T12:02:28.148 INFO:tasks.cephadm.ceph_manager.ceph:need seq 158913790016 got 158913790022 for osd.5 2024-05-26T12:02:28.149 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:28.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:28 smithi156 ceph-mon[32025]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:28.777 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:28 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2064595086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-26T12:02:28.810 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738456 got 34359738463 for osd.0 2024-05-26T12:02:28.810 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:28 smithi186 ceph-mon[36412]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:28.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:28 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2064595086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-26T12:02:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:28 smithi192 ceph-mon[36545]: pgmap v353: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:29.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:28 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2064595086' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-26T12:02:29.401 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411408 got 77309411414 for osd.2 2024-05-26T12:02:29.401 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:30.917 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:30 smithi156 ceph-mon[32025]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 -- 172.21.15.156:0/3066541088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc108920 msgr2=0x7fdccc1104a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 --2- 172.21.15.156:0/3066541088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc108920 0x7fdccc1104a0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdcc400b0d0 tx=0x7fdcc400b3e0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 -- 172.21.15.156:0/3066541088 shutdown_connections 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 --2- 172.21.15.156:0/3066541088 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc108920 0x7fdccc1104a0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 --2- 172.21.15.156:0/3066541088 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc103850 0x7fdccc103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 --2- 172.21.15.156:0/3066541088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc104200 0x7fdccc1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.939 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 -- 172.21.15.156:0/3066541088 >> 172.21.15.156:0/3066541088 conn(0x7fdccc07a620 msgr2=0x7fdccc07aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 -- 172.21.15.156:0/3066541088 shutdown_connections 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.916+0000 7fdcd4696700 1 -- 172.21.15.156:0/3066541088 wait complete. 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 Processor -- start 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 -- start start 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc104200 0x7fdccc19fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc108920 0x7fdccc199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc1137e0 con 0x7fdccc103850 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc113660 con 0x7fdccc108920 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd4696700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdccc113960 con 0x7fdccc104200 2024-05-26T12:02:30.940 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55692/0 (socket says 172.21.15.156:55692) 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 -- 172.21.15.156:0/290548200 learned_addr learned my addr 172.21.15.156:0/290548200 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2c33700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc108920 0x7fdccc199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd1c31700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc104200 0x7fdccc19fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 -- 172.21.15.156:0/290548200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc104200 msgr2=0x7fdccc19fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc104200 0x7fdccc19fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 -- 172.21.15.156:0/290548200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc108920 msgr2=0x7fdccc199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc108920 0x7fdccc199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.917+0000 7fdcd2432700 1 -- 172.21.15.156:0/290548200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdccc19a200 con 0x7fdccc103850 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcd2432700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00b530 tx=0x7fdcbc00b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc015020 con 0x7fdccc103850 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdccc19a4f0 con 0x7fdccc103850 2024-05-26T12:02:30.941 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdcbc011ce0 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcbc010dd0 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.918+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdccc1a67b0 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.919+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdccc10bb60 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.920+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7fdcbc018020 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.921+0000 7fdcc37fe700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 0x7fdcb80b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.921+0000 7fdcd1c31700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 0x7fdcb80b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.921+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fdcbc0e1fe0 con 0x7fdccc103850 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.921+0000 7fdcd1c31700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 0x7fdcb80b4dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdccc103210 tx=0x7fdcc8009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:30.942 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:30.925+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdcbc01f020 con 0x7fdccc103850 2024-05-26T12:02:30.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:30 smithi186 ceph-mon[36412]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:02:31.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:30 smithi192 ceph-mon[36545]: pgmap v354: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:02:31.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.138+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fdccc19afd0 con 0x7fdccc103850 2024-05-26T12:02:31.139 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.138+0000 7fdcc37fe700 1 -- 172.21.15.156:0/290548200 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fdcbc0a9c20 con 0x7fdccc103850 2024-05-26T12:02:31.139 INFO:teuthology.orchestra.run.smithi156.stdout:111669149777 2024-05-26T12:02:31.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 msgr2=0x7fdcb80b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:31.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 0x7fdcb80b4dd0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fdccc103210 tx=0x7fdcc8009190 comp rx=0 tx=0).stop 2024-05-26T12:02:31.141 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 msgr2=0x7fdccc19f8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdcbc00b530 tx=0x7fdcbc00b840 comp rx=0 tx=0).stop 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 shutdown_connections 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdcb80b2910 0x7fdcb80b4dd0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdccc108920 0x7fdccc199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdccc103850 0x7fdccc19f8e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 --2- 172.21.15.156:0/290548200 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdccc104200 0x7fdccc19fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 >> 172.21.15.156:0/290548200 conn(0x7fdccc07a620 msgr2=0x7fdccc101950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.140+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 shutdown_connections 2024-05-26T12:02:31.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:31.141+0000 7fdcd4696700 1 -- 172.21.15.156:0/290548200 wait complete. 2024-05-26T12:02:31.781 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:31 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/290548200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-26T12:02:31.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149772 got 111669149777 for osd.3 2024-05-26T12:02:31.809 DEBUG:teuthology.parallel:result is None 2024-05-26T12:02:31.809 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-26T12:02:31.809 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-26T12:02:31.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:31 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/290548200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-26T12:02:32.054 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:32.071 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:31 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/290548200' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-26T12:02:32.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:32 smithi156 ceph-mon[32025]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:32.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:32 smithi186 ceph-mon[36412]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:33.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:32 smithi192 ceph-mon[36545]: pgmap v355: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:02:33.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 -- 172.21.15.156:0/2745503980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc1084f0 msgr2=0x7ff5fc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:33.751 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 --2- 172.21.15.156:0/2745503980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc1088d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0009990 tx=0x7ff5f0009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 -- 172.21.15.156:0/2745503980 shutdown_connections 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 --2- 172.21.15.156:0/2745503980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc102e20 0x7ff5fc10f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 --2- 172.21.15.156:0/2745503980 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc1088d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 --2- 172.21.15.156:0/2745503980 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc1028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 -- 172.21.15.156:0/2745503980 >> 172.21.15.156:0/2745503980 conn(0x7ff5fc0794e0 msgr2=0x7ff5fc0798f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 -- 172.21.15.156:0/2745503980 shutdown_connections 2024-05-26T12:02:33.752 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.750+0000 7ff6030c5700 1 -- 172.21.15.156:0/2745503980 wait complete. 2024-05-26T12:02:33.752 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:33 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 Processor -- start 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 -- start start 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc19f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc112620 con 0x7ff5fc102e20 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc1124a0 con 0x7ff5fc1084f0 2024-05-26T12:02:33.754 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff6030c5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff5fc1127a0 con 0x7ff5fc102460 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff5fbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff5fbfff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:55706/0 (socket says 172.21.15.156:55706) 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff5fbfff700 1 -- 172.21.15.156:0/753344755 learned_addr learned my addr 172.21.15.156:0/753344755 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff601662700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.751+0000 7ff600e61700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc19f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 -- 172.21.15.156:0/753344755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 msgr2=0x7ff5fc19f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc19f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 -- 172.21.15.156:0/753344755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 msgr2=0x7ff5fc199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 -- 172.21.15.156:0/753344755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc19a170 con 0x7ff5fc102e20 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff601662700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:02:33.755 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff600e61700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5fbfff700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800e9c0 tx=0x7ff5e800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800cbd0 con 0x7ff5fc102e20 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff5e800cd70 con 0x7ff5fc102e20 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc19a460 con 0x7ff5fc102e20 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8005610 con 0x7ff5fc102e20 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.752+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5fc0ff3c0 con 0x7ff5fc102e20 2024-05-26T12:02:33.756 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.754+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5fc06a230 con 0x7ff5fc102e20 2024-05-26T12:02:33.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.755+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7ff5e8004750 con 0x7ff5fc102e20 2024-05-26T12:02:33.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.755+0000 7ff5f9ffb700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 0x7ff5ec0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:33.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.756+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7ff5e80e2760 con 0x7ff5fc102e20 2024-05-26T12:02:33.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.756+0000 7ff600e61700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 0x7ff5ec0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:33.757 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.756+0000 7ff600e61700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 0x7ff5ec0b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006010 tx=0x7ff5f001a040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:33.761 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.761+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff5e80aa2f0 con 0x7ff5fc102e20 2024-05-26T12:02:33.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:33 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:02:33.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:33.995+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff5fc06bc70 con 0x7ff5ec0b2910 2024-05-26T12:02:34.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.002+0000 7ff5f9ffb700 1 -- 172.21.15.156:0/753344755 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259020 (secure 0 0 0) 0x7ff5f00663e0 con 0x7ff5ec0b2910 2024-05-26T12:02:34.007 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 msgr2=0x7ff5ec0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 0x7ff5ec0b4dd0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0006010 tx=0x7ff5f001a040 comp rx=0 tx=0).stop 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 msgr2=0x7ff5fc19fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ff5e800e9c0 tx=0x7ff5e800ecd0 comp rx=0 tx=0).stop 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 shutdown_connections 2024-05-26T12:02:34.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff5ec0b2910 0x7ff5ec0b4dd0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5fc1084f0 0x7ff5fc199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff5fc102460 0x7ff5fc19f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 --2- 172.21.15.156:0/753344755 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff5fc102e20 0x7ff5fc19fd90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 >> 172.21.15.156:0/753344755 conn(0x7ff5fc0794e0 msgr2=0x7ff5fc101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 shutdown_connections 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:34.006+0000 7ff6030c5700 1 -- 172.21.15.156:0/753344755 wait complete. 2024-05-26T12:02:34.010 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-05-26T12:02:34.049 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:33 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:02:34.768 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:34 smithi156 ceph-mon[32025]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:34.798 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":356,"stamp":"2024-05-26T12:02:32.824624+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657177,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1085,"num_read_kb":635,"num_write":369,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2980013,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356240,"kb_used_data":11168,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585584,"statfs":{"total":1151940427776,"available":1151575638016,"internally_reserved":0,"allocated":11436032,"data_stored":7308039,"data_compressed":251796,"data_compressed_allocated":339968,"data_compressed_original":2187264,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":110,"num_read_kb":55,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002966"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640541+0000","last_change":"2024-05-26T12:00:32.620823+0000","last_active":"2024-05-26T12:02:08.640541+0000","last_peered":"2024-05-26T12:02:08.640541+0000","last_clean":"2024-05-26T12:02:08.640541+0000","last_became_active":"2024-05-26T12:00:32.619171+0000","last_became_peered":"2024-05-26T12:00:32.619171+0000","last_unstale":"2024-05-26T12:02:08.640541+0000","last_undegraded":"2024-05-26T12:02:08.640541+0000","last_fullsized":"2024-05-26T12:02:08.640541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:57:59.459768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835008+0000","last_change":"2024-05-26T12:00:32.618888+0000","last_active":"2024-05-26T12:02:08.835008+0000","last_peered":"2024-05-26T12:02:08.835008+0000","last_clean":"2024-05-26T12:02:08.835008+0000","last_became_active":"2024-05-26T12:00:32.618235+0000","last_became_peered":"2024-05-26T12:00:32.618235+0000","last_unstale":"2024-05-26T12:02:08.835008+0000","last_undegraded":"2024-05-26T12:02:08.835008+0000","last_fullsized":"2024-05-26T12:02:08.835008+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:56:08.852594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.834893+0000","last_change":"2024-05-26T12:00:32.613694+0000","last_active":"2024-05-26T12:02:08.834893+0000","last_peered":"2024-05-26T12:02:08.834893+0000","last_clean":"2024-05-26T12:02:08.834893+0000","last_became_active":"2024-05-26T12:00:32.613389+0000","last_became_peered":"2024-05-26T12:00:32.613389+0000","last_unstale":"2024-05-26T12:02:08.834893+0000","last_undegraded":"2024-05-26T12:02:08.834893+0000","last_fullsized":"2024-05-26T12:02:08.834893+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:04.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621256+0000","last_change":"2024-05-26T12:00:32.620149+0000","last_active":"2024-05-26T12:02:08.621256+0000","last_peered":"2024-05-26T12:02:08.621256+0000","last_clean":"2024-05-26T12:02:08.621256+0000","last_became_active":"2024-05-26T12:00:32.614534+0000","last_became_peered":"2024-05-26T12:00:32.614534+0000","last_unstale":"2024-05-26T12:02:08.621256+0000","last_undegraded":"2024-05-26T12:02:08.621256+0000","last_fullsized":"2024-05-26T12:02:08.621256+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:24:41.144826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639969+0000","last_change":"2024-05-26T12:00:32.605841+0000","last_active":"2024-05-26T12:02:08.639969+0000","last_peered":"2024-05-26T12:02:08.639969+0000","last_clean":"2024-05-26T12:02:08.639969+0000","last_became_active":"2024-05-26T12:00:32.605438+0000","last_became_peered":"2024-05-26T12:00:32.605438+0000","last_unstale":"2024-05-26T12:02:08.639969+0000","last_undegraded":"2024-05-26T12:02:08.639969+0000","last_fullsized":"2024-05-26T12:02:08.639969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:05:01.341398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579377+0000","last_change":"2024-05-26T12:00:32.625270+0000","last_active":"2024-05-26T12:02:08.579377+0000","last_peered":"2024-05-26T12:02:08.579377+0000","last_clean":"2024-05-26T12:02:08.579377+0000","last_became_active":"2024-05-26T12:00:32.624892+0000","last_became_peered":"2024-05-26T12:00:32.624892+0000","last_unstale":"2024-05-26T12:02:08.579377+0000","last_undegraded":"2024-05-26T12:02:08.579377+0000","last_fullsized":"2024-05-26T12:02:08.579377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:44.581793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639797+0000","last_change":"2024-05-26T12:00:32.613286+0000","last_active":"2024-05-26T12:02:08.639797+0000","last_peered":"2024-05-26T12:02:08.639797+0000","last_clean":"2024-05-26T12:02:08.639797+0000","last_became_active":"2024-05-26T12:00:32.612896+0000","last_became_peered":"2024-05-26T12:00:32.612896+0000","last_unstale":"2024-05-26T12:02:08.639797+0000","last_undegraded":"2024-05-26T12:02:08.639797+0000","last_fullsized":"2024-05-26T12:02:08.639797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:34:29.796529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536725+0000","last_change":"2024-05-26T12:00:32.620975+0000","last_active":"2024-05-26T12:02:08.536725+0000","last_peered":"2024-05-26T12:02:08.536725+0000","last_clean":"2024-05-26T12:02:08.536725+0000","last_became_active":"2024-05-26T12:00:32.618576+0000","last_became_peered":"2024-05-26T12:00:32.618576+0000","last_unstale":"2024-05-26T12:02:08.536725+0000","last_undegraded":"2024-05-26T12:02:08.536725+0000","last_fullsized":"2024-05-26T12:02:08.536725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:51:04.187910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538637+0000","last_change":"2024-05-26T12:00:32.618617+0000","last_active":"2024-05-26T12:02:08.538637+0000","last_peered":"2024-05-26T12:02:08.538637+0000","last_clean":"2024-05-26T12:02:08.538637+0000","last_became_active":"2024-05-26T12:00:32.617354+0000","last_became_peered":"2024-05-26T12:00:32.617354+0000","last_unstale":"2024-05-26T12:02:08.538637+0000","last_undegraded":"2024-05-26T12:02:08.538637+0000","last_fullsized":"2024-05-26T12:02:08.538637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:04:22.469952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579758+0000","last_change":"2024-05-26T12:00:32.618169+0000","last_active":"2024-05-26T12:02:08.579758+0000","last_peered":"2024-05-26T12:02:08.579758+0000","last_clean":"2024-05-26T12:02:08.579758+0000","last_became_active":"2024-05-26T12:00:32.617709+0000","last_became_peered":"2024-05-26T12:00:32.617709+0000","last_unstale":"2024-05-26T12:02:08.579758+0000","last_undegraded":"2024-05-26T12:02:08.579758+0000","last_fullsized":"2024-05-26T12:02:08.579758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:21.801512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639958+0000","last_change":"2024-05-26T12:00:32.623233+0000","last_active":"2024-05-26T12:02:08.639958+0000","last_peered":"2024-05-26T12:02:08.639958+0000","last_clean":"2024-05-26T12:02:08.639958+0000","last_became_active":"2024-05-26T12:00:32.622483+0000","last_became_peered":"2024-05-26T12:00:32.622483+0000","last_unstale":"2024-05-26T12:02:08.639958+0000","last_undegraded":"2024-05-26T12:02:08.639958+0000","last_fullsized":"2024-05-26T12:02:08.639958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:14.784275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621365+0000","last_change":"2024-05-26T12:00:32.614575+0000","last_active":"2024-05-26T12:02:08.621365+0000","last_peered":"2024-05-26T12:02:08.621365+0000","last_clean":"2024-05-26T12:02:08.621365+0000","last_became_active":"2024-05-26T12:00:32.614020+0000","last_became_peered":"2024-05-26T12:00:32.614020+0000","last_unstale":"2024-05-26T12:02:08.621365+0000","last_undegraded":"2024-05-26T12:02:08.621365+0000","last_fullsized":"2024-05-26T12:02:08.621365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:33:05.747419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618727+0000","last_change":"2024-05-26T12:00:32.616624+0000","last_active":"2024-05-26T12:02:08.618727+0000","last_peered":"2024-05-26T12:02:08.618727+0000","last_clean":"2024-05-26T12:02:08.618727+0000","last_became_active":"2024-05-26T12:00:32.616280+0000","last_became_peered":"2024-05-26T12:00:32.616280+0000","last_unstale":"2024-05-26T12:02:08.618727+0000","last_undegraded":"2024-05-26T12:02:08.618727+0000","last_fullsized":"2024-05-26T12:02:08.618727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:45:02.466249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835014+0000","last_change":"2024-05-26T12:00:32.617058+0000","last_active":"2024-05-26T12:02:08.835014+0000","last_peered":"2024-05-26T12:02:08.835014+0000","last_clean":"2024-05-26T12:02:08.835014+0000","last_became_active":"2024-05-26T12:00:32.614651+0000","last_became_peered":"2024-05-26T12:00:32.614651+0000","last_unstale":"2024-05-26T12:02:08.835014+0000","last_undegraded":"2024-05-26T12:02:08.835014+0000","last_fullsized":"2024-05-26T12:02:08.835014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:38:50.234745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835005+0000","last_change":"2024-05-26T12:01:07.950685+0000","last_active":"2024-05-26T12:02:08.835005+0000","last_peered":"2024-05-26T12:02:08.835005+0000","last_clean":"2024-05-26T12:02:08.835005+0000","last_became_active":"2024-05-26T12:01:07.950383+0000","last_became_peered":"2024-05-26T12:01:07.950383+0000","last_unstale":"2024-05-26T12:02:08.835005+0000","last_undegraded":"2024-05-26T12:02:08.835005+0000","last_fullsized":"2024-05-26T12:02:08.835005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:02:42.832274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538100+0000","last_change":"2024-05-26T12:00:32.618850+0000","last_active":"2024-05-26T12:02:08.538100+0000","last_peered":"2024-05-26T12:02:08.538100+0000","last_clean":"2024-05-26T12:02:08.538100+0000","last_became_active":"2024-05-26T12:00:32.618494+0000","last_became_peered":"2024-05-26T12:00:32.618494+0000","last_unstale":"2024-05-26T12:02:08.538100+0000","last_undegraded":"2024-05-26T12:02:08.538100+0000","last_fullsized":"2024-05-26T12:02:08.538100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:23:43.555380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640263+0000","last_change":"2024-05-26T12:00:32.621515+0000","last_active":"2024-05-26T12:02:08.640263+0000","last_peered":"2024-05-26T12:02:08.640263+0000","last_clean":"2024-05-26T12:02:08.640263+0000","last_became_active":"2024-05-26T12:00:32.621091+0000","last_became_peered":"2024-05-26T12:00:32.621091+0000","last_unstale":"2024-05-26T12:02:08.640263+0000","last_undegraded":"2024-05-26T12:02:08.640263+0000","last_fullsized":"2024-05-26T12:02:08.640263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:32:07.614386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538389+0000","last_change":"2024-05-26T12:00:32.612371+0000","last_active":"2024-05-26T12:02:08.538389+0000","last_peered":"2024-05-26T12:02:08.538389+0000","last_clean":"2024-05-26T12:02:08.538389+0000","last_became_active":"2024-05-26T12:00:32.612082+0000","last_became_peered":"2024-05-26T12:00:32.612082+0000","last_unstale":"2024-05-26T12:02:08.538389+0000","last_undegraded":"2024-05-26T12:02:08.538389+0000","last_fullsized":"2024-05-26T12:02:08.538389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:25:15.850251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537943+0000","last_change":"2024-05-26T12:00:32.611421+0000","last_active":"2024-05-26T12:02:08.537943+0000","last_peered":"2024-05-26T12:02:08.537943+0000","last_clean":"2024-05-26T12:02:08.537943+0000","last_became_active":"2024-05-26T12:00:32.609716+0000","last_became_peered":"2024-05-26T12:00:32.609716+0000","last_unstale":"2024-05-26T12:02:08.537943+0000","last_undegraded":"2024-05-26T12:02:08.537943+0000","last_fullsized":"2024-05-26T12:02:08.537943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:55.197945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":141,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:27.679431+0000","last_change":"2024-05-26T12:00:28.578034+0000","last_active":"2024-05-26T12:02:27.679431+0000","last_peered":"2024-05-26T12:02:27.679431+0000","last_clean":"2024-05-26T12:02:27.679431+0000","last_became_active":"2024-05-26T12:00:28.577682+0000","last_became_peered":"2024-05-26T12:00:28.577682+0000","last_unstale":"2024-05-26T12:02:27.679431+0000","last_undegraded":"2024-05-26T12:02:27.679431+0000","last_fullsized":"2024-05-26T12:02:27.679431+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:27:11.135886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":98,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537064+0000","last_change":"2024-05-26T12:00:32.609857+0000","last_active":"2024-05-26T12:02:08.537064+0000","last_peered":"2024-05-26T12:02:08.537064+0000","last_clean":"2024-05-26T12:02:08.537064+0000","last_became_active":"2024-05-26T12:00:32.609380+0000","last_became_peered":"2024-05-26T12:00:32.609380+0000","last_unstale":"2024-05-26T12:02:08.537064+0000","last_undegraded":"2024-05-26T12:02:08.537064+0000","last_fullsized":"2024-05-26T12:02:08.537064+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:39:04.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537252+0000","last_change":"2024-05-26T12:00:28.671940+0000","last_active":"2024-05-26T12:02:08.537252+0000","last_peered":"2024-05-26T12:02:08.537252+0000","last_clean":"2024-05-26T12:02:08.537252+0000","last_became_active":"2024-05-26T12:00:28.671602+0000","last_became_peered":"2024-05-26T12:00:28.671602+0000","last_unstale":"2024-05-26T12:02:08.537252+0000","last_undegraded":"2024-05-26T12:02:08.537252+0000","last_fullsized":"2024-05-26T12:02:08.537252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:38:52.069136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.834962+0000","last_change":"2024-05-26T12:00:32.613900+0000","last_active":"2024-05-26T12:02:08.834962+0000","last_peered":"2024-05-26T12:02:08.834962+0000","last_clean":"2024-05-26T12:02:08.834962+0000","last_became_active":"2024-05-26T12:00:32.610264+0000","last_became_peered":"2024-05-26T12:00:32.610264+0000","last_unstale":"2024-05-26T12:02:08.834962+0000","last_undegraded":"2024-05-26T12:02:08.834962+0000","last_fullsized":"2024-05-26T12:02:08.834962+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:28:58.712711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640747+0000","last_change":"2024-05-26T12:01:08.483676+0000","last_active":"2024-05-26T12:02:08.640747+0000","last_peered":"2024-05-26T12:02:08.640747+0000","last_clean":"2024-05-26T12:02:08.640747+0000","last_became_active":"2024-05-26T12:01:08.471437+0000","last_became_peered":"2024-05-26T12:01:08.471437+0000","last_unstale":"2024-05-26T12:02:08.640747+0000","last_undegraded":"2024-05-26T12:02:08.640747+0000","last_fullsized":"2024-05-26T12:02:08.640747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:43:38.663368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538279+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:02:08.538279+0000","last_peered":"2024-05-26T12:02:08.538279+0000","last_clean":"2024-05-26T12:02:08.538279+0000","last_became_active":"2024-05-26T12:00:32.617702+0000","last_became_peered":"2024-05-26T12:00:32.617702+0000","last_unstale":"2024-05-26T12:02:08.538279+0000","last_undegraded":"2024-05-26T12:02:08.538279+0000","last_fullsized":"2024-05-26T12:02:08.538279+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:25:06.527677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640790+0000","last_change":"2024-05-26T12:00:28.575648+0000","last_active":"2024-05-26T12:02:08.640790+0000","last_peered":"2024-05-26T12:02:08.640790+0000","last_clean":"2024-05-26T12:02:08.640790+0000","last_became_active":"2024-05-26T12:00:28.575170+0000","last_became_peered":"2024-05-26T12:00:28.575170+0000","last_unstale":"2024-05-26T12:02:08.640790+0000","last_undegraded":"2024-05-26T12:02:08.640790+0000","last_fullsized":"2024-05-26T12:02:08.640790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:09:29.213605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582175+0000","last_change":"2024-05-26T12:00:32.617825+0000","last_active":"2024-05-26T12:02:08.582175+0000","last_peered":"2024-05-26T12:02:08.582175+0000","last_clean":"2024-05-26T12:02:08.582175+0000","last_became_active":"2024-05-26T12:00:32.617517+0000","last_became_peered":"2024-05-26T12:00:32.617517+0000","last_unstale":"2024-05-26T12:02:08.582175+0000","last_undegraded":"2024-05-26T12:02:08.582175+0000","last_fullsized":"2024-05-26T12:02:08.582175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:36:33.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":134,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:27.642952+0000","last_change":"2024-05-26T12:00:28.584639+0000","last_active":"2024-05-26T12:02:27.642952+0000","last_peered":"2024-05-26T12:02:27.642952+0000","last_clean":"2024-05-26T12:02:27.642952+0000","last_became_active":"2024-05-26T12:00:28.583063+0000","last_became_peered":"2024-05-26T12:00:28.583063+0000","last_unstale":"2024-05-26T12:02:27.642952+0000","last_undegraded":"2024-05-26T12:02:27.642952+0000","last_fullsized":"2024-05-26T12:02:27.642952+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:01:25.804547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":99,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639966+0000","last_change":"2024-05-26T12:00:32.609117+0000","last_active":"2024-05-26T12:02:08.639966+0000","last_peered":"2024-05-26T12:02:08.639966+0000","last_clean":"2024-05-26T12:02:08.639966+0000","last_became_active":"2024-05-26T12:00:32.608770+0000","last_became_peered":"2024-05-26T12:00:32.608770+0000","last_unstale":"2024-05-26T12:02:08.639966+0000","last_undegraded":"2024-05-26T12:02:08.639966+0000","last_fullsized":"2024-05-26T12:02:08.639966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:19:57.422101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579332+0000","last_change":"2024-05-26T12:01:07.954880+0000","last_active":"2024-05-26T12:02:08.579332+0000","last_peered":"2024-05-26T12:02:08.579332+0000","last_clean":"2024-05-26T12:02:08.579332+0000","last_became_active":"2024-05-26T12:01:07.949506+0000","last_became_peered":"2024-05-26T12:01:07.949506+0000","last_unstale":"2024-05-26T12:02:08.579332+0000","last_undegraded":"2024-05-26T12:02:08.579332+0000","last_fullsized":"2024-05-26T12:02:08.579332+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:40:37.041619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582377+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:02:08.582377+0000","last_peered":"2024-05-26T12:02:08.582377+0000","last_clean":"2024-05-26T12:02:08.582377+0000","last_became_active":"2024-05-26T12:00:32.615691+0000","last_became_peered":"2024-05-26T12:00:32.615691+0000","last_unstale":"2024-05-26T12:02:08.582377+0000","last_undegraded":"2024-05-26T12:02:08.582377+0000","last_fullsized":"2024-05-26T12:02:08.582377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:58:26.245149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619059+0000","last_change":"2024-05-26T12:00:28.571893+0000","last_active":"2024-05-26T12:02:08.619059+0000","last_peered":"2024-05-26T12:02:08.619059+0000","last_clean":"2024-05-26T12:02:08.619059+0000","last_became_active":"2024-05-26T12:00:28.571502+0000","last_became_peered":"2024-05-26T12:00:28.571502+0000","last_unstale":"2024-05-26T12:02:08.619059+0000","last_undegraded":"2024-05-26T12:02:08.619059+0000","last_fullsized":"2024-05-26T12:02:08.619059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:08:01.040087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536788+0000","last_change":"2024-05-26T12:00:32.616575+0000","last_active":"2024-05-26T12:02:08.536788+0000","last_peered":"2024-05-26T12:02:08.536788+0000","last_clean":"2024-05-26T12:02:08.536788+0000","last_became_active":"2024-05-26T12:00:32.616261+0000","last_became_peered":"2024-05-26T12:00:32.616261+0000","last_unstale":"2024-05-26T12:02:08.536788+0000","last_undegraded":"2024-05-26T12:02:08.536788+0000","last_fullsized":"2024-05-26T12:02:08.536788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:06:50.012838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581855+0000","last_change":"2024-05-26T12:00:28.584129+0000","last_active":"2024-05-26T12:02:08.581855+0000","last_peered":"2024-05-26T12:02:08.581855+0000","last_clean":"2024-05-26T12:02:08.581855+0000","last_became_active":"2024-05-26T12:00:28.583046+0000","last_became_peered":"2024-05-26T12:00:28.583046+0000","last_unstale":"2024-05-26T12:02:08.581855+0000","last_undegraded":"2024-05-26T12:02:08.581855+0000","last_fullsized":"2024-05-26T12:02:08.581855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:47.635696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538138+0000","last_change":"2024-05-26T12:00:32.610148+0000","last_active":"2024-05-26T12:02:08.538138+0000","last_peered":"2024-05-26T12:02:08.538138+0000","last_clean":"2024-05-26T12:02:08.538138+0000","last_became_active":"2024-05-26T12:00:32.609849+0000","last_became_peered":"2024-05-26T12:00:32.609849+0000","last_unstale":"2024-05-26T12:02:08.538138+0000","last_undegraded":"2024-05-26T12:02:08.538138+0000","last_fullsized":"2024-05-26T12:02:08.538138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:37:08.938164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":230,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:29.645517+0000","last_change":"2024-05-26T12:00:28.578831+0000","last_active":"2024-05-26T12:02:29.645517+0000","last_peered":"2024-05-26T12:02:29.645517+0000","last_clean":"2024-05-26T12:02:29.645517+0000","last_became_active":"2024-05-26T12:00:28.578484+0000","last_became_peered":"2024-05-26T12:00:28.578484+0000","last_unstale":"2024-05-26T12:02:29.645517+0000","last_undegraded":"2024-05-26T12:02:29.645517+0000","last_fullsized":"2024-05-26T12:02:29.645517+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:11:07.127948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":196,"num_read_kb":196,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538626+0000","last_change":"2024-05-26T12:00:32.618151+0000","last_active":"2024-05-26T12:02:08.538626+0000","last_peered":"2024-05-26T12:02:08.538626+0000","last_clean":"2024-05-26T12:02:08.538626+0000","last_became_active":"2024-05-26T12:00:32.616835+0000","last_became_peered":"2024-05-26T12:00:32.616835+0000","last_unstale":"2024-05-26T12:02:08.538626+0000","last_undegraded":"2024-05-26T12:02:08.538626+0000","last_fullsized":"2024-05-26T12:02:08.538626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:17:57.682573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640171+0000","last_change":"2024-05-26T12:00:28.579624+0000","last_active":"2024-05-26T12:02:08.640171+0000","last_peered":"2024-05-26T12:02:08.640171+0000","last_clean":"2024-05-26T12:02:08.640171+0000","last_became_active":"2024-05-26T12:00:28.578691+0000","last_became_peered":"2024-05-26T12:00:28.578691+0000","last_unstale":"2024-05-26T12:02:08.640171+0000","last_undegraded":"2024-05-26T12:02:08.640171+0000","last_fullsized":"2024-05-26T12:02:08.640171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:38:12.319875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835233+0000","last_change":"2024-05-26T12:00:32.610573+0000","last_active":"2024-05-26T12:02:08.835233+0000","last_peered":"2024-05-26T12:02:08.835233+0000","last_clean":"2024-05-26T12:02:08.835233+0000","last_became_active":"2024-05-26T12:00:32.609824+0000","last_became_peered":"2024-05-26T12:00:32.609824+0000","last_unstale":"2024-05-26T12:02:08.835233+0000","last_undegraded":"2024-05-26T12:02:08.835233+0000","last_fullsized":"2024-05-26T12:02:08.835233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:56.631874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582284+0000","last_change":"2024-05-26T12:00:28.672441+0000","last_active":"2024-05-26T12:02:08.582284+0000","last_peered":"2024-05-26T12:02:08.582284+0000","last_clean":"2024-05-26T12:02:08.582284+0000","last_became_active":"2024-05-26T12:00:28.672039+0000","last_became_peered":"2024-05-26T12:00:28.672039+0000","last_unstale":"2024-05-26T12:02:08.582284+0000","last_undegraded":"2024-05-26T12:02:08.582284+0000","last_fullsized":"2024-05-26T12:02:08.582284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:14:09.757458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538369+0000","last_change":"2024-05-26T12:00:32.612289+0000","last_active":"2024-05-26T12:02:08.538369+0000","last_peered":"2024-05-26T12:02:08.538369+0000","last_clean":"2024-05-26T12:02:08.538369+0000","last_became_active":"2024-05-26T12:00:32.611930+0000","last_became_peered":"2024-05-26T12:00:32.611930+0000","last_unstale":"2024-05-26T12:02:08.538369+0000","last_undegraded":"2024-05-26T12:02:08.538369+0000","last_fullsized":"2024-05-26T12:02:08.538369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:05:55.351242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'31","reported_seq":284,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:29.616901+0000","last_change":"2024-05-26T12:01:08.493089+0000","last_active":"2024-05-26T12:02:29.616901+0000","last_peered":"2024-05-26T12:02:29.616901+0000","last_clean":"2024-05-26T12:02:29.616901+0000","last_became_active":"2024-05-26T12:01:08.471966+0000","last_became_peered":"2024-05-26T12:01:08.471966+0000","last_unstale":"2024-05-26T12:02:29.616901+0000","last_undegraded":"2024-05-26T12:02:29.616901+0000","last_fullsized":"2024-05-26T12:02:29.616901+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:51.413474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59762,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":196,"num_read_kb":150,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":28083,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640426+0000","last_change":"2024-05-26T12:00:32.614557+0000","last_active":"2024-05-26T12:02:08.640426+0000","last_peered":"2024-05-26T12:02:08.640426+0000","last_clean":"2024-05-26T12:02:08.640426+0000","last_became_active":"2024-05-26T12:00:32.613806+0000","last_became_peered":"2024-05-26T12:00:32.613806+0000","last_unstale":"2024-05-26T12:02:08.640426+0000","last_undegraded":"2024-05-26T12:02:08.640426+0000","last_fullsized":"2024-05-26T12:02:08.640426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:11:11.279479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538147+0000","last_change":"2024-05-26T12:00:28.577551+0000","last_active":"2024-05-26T12:02:08.538147+0000","last_peered":"2024-05-26T12:02:08.538147+0000","last_clean":"2024-05-26T12:02:08.538147+0000","last_became_active":"2024-05-26T12:00:28.577203+0000","last_became_peered":"2024-05-26T12:00:28.577203+0000","last_unstale":"2024-05-26T12:02:08.538147+0000","last_undegraded":"2024-05-26T12:02:08.538147+0000","last_fullsized":"2024-05-26T12:02:08.538147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:22:54.906432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618745+0000","last_change":"2024-05-26T12:00:32.612029+0000","last_active":"2024-05-26T12:02:08.618745+0000","last_peered":"2024-05-26T12:02:08.618745+0000","last_clean":"2024-05-26T12:02:08.618745+0000","last_became_active":"2024-05-26T12:00:32.611660+0000","last_became_peered":"2024-05-26T12:00:32.611660+0000","last_unstale":"2024-05-26T12:02:08.618745+0000","last_undegraded":"2024-05-26T12:02:08.618745+0000","last_fullsized":"2024-05-26T12:02:08.618745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:15:27.278693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640375+0000","last_change":"2024-05-26T12:01:07.960293+0000","last_active":"2024-05-26T12:02:08.640375+0000","last_peered":"2024-05-26T12:02:08.640375+0000","last_clean":"2024-05-26T12:02:08.640375+0000","last_became_active":"2024-05-26T12:01:07.947178+0000","last_became_peered":"2024-05-26T12:01:07.947178+0000","last_unstale":"2024-05-26T12:02:08.640375+0000","last_undegraded":"2024-05-26T12:02:08.640375+0000","last_fullsized":"2024-05-26T12:02:08.640375+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:28:23.807776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640288+0000","last_change":"2024-05-26T12:01:07.948672+0000","last_active":"2024-05-26T12:02:08.640288+0000","last_peered":"2024-05-26T12:02:08.640288+0000","last_clean":"2024-05-26T12:02:08.640288+0000","last_became_active":"2024-05-26T12:01:07.948285+0000","last_became_peered":"2024-05-26T12:01:07.948285+0000","last_unstale":"2024-05-26T12:02:08.640288+0000","last_undegraded":"2024-05-26T12:02:08.640288+0000","last_fullsized":"2024-05-26T12:02:08.640288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:49:39.764352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":135,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:28.658322+0000","last_change":"2024-05-26T12:00:28.578507+0000","last_active":"2024-05-26T12:02:28.658322+0000","last_peered":"2024-05-26T12:02:28.658322+0000","last_clean":"2024-05-26T12:02:28.658322+0000","last_became_active":"2024-05-26T12:00:28.578131+0000","last_became_peered":"2024-05-26T12:00:28.578131+0000","last_unstale":"2024-05-26T12:02:28.658322+0000","last_undegraded":"2024-05-26T12:02:28.658322+0000","last_fullsized":"2024-05-26T12:02:28.658322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:14:52.015296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":101,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639749+0000","last_change":"2024-05-26T12:00:32.618061+0000","last_active":"2024-05-26T12:02:08.639749+0000","last_peered":"2024-05-26T12:02:08.639749+0000","last_clean":"2024-05-26T12:02:08.639749+0000","last_became_active":"2024-05-26T12:00:32.617320+0000","last_became_peered":"2024-05-26T12:00:32.617320+0000","last_unstale":"2024-05-26T12:02:08.639749+0000","last_undegraded":"2024-05-26T12:02:08.639749+0000","last_fullsized":"2024-05-26T12:02:08.639749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:57:33.972120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640264+0000","last_change":"2024-05-26T12:00:28.672890+0000","last_active":"2024-05-26T12:02:08.640264+0000","last_peered":"2024-05-26T12:02:08.640264+0000","last_clean":"2024-05-26T12:02:08.640264+0000","last_became_active":"2024-05-26T12:00:28.672523+0000","last_became_peered":"2024-05-26T12:00:28.672523+0000","last_unstale":"2024-05-26T12:02:08.640264+0000","last_undegraded":"2024-05-26T12:02:08.640264+0000","last_fullsized":"2024-05-26T12:02:08.640264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:46:00.281960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581992+0000","last_change":"2024-05-26T12:00:32.618020+0000","last_active":"2024-05-26T12:02:08.581992+0000","last_peered":"2024-05-26T12:02:08.581992+0000","last_clean":"2024-05-26T12:02:08.581992+0000","last_became_active":"2024-05-26T12:00:32.617147+0000","last_became_peered":"2024-05-26T12:00:32.617147+0000","last_unstale":"2024-05-26T12:02:08.581992+0000","last_undegraded":"2024-05-26T12:02:08.581992+0000","last_fullsized":"2024-05-26T12:02:08.581992+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:54:06.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640473+0000","last_change":"2024-05-26T12:00:28.573265+0000","last_active":"2024-05-26T12:02:08.640473+0000","last_peered":"2024-05-26T12:02:08.640473+0000","last_clean":"2024-05-26T12:02:08.640473+0000","last_became_active":"2024-05-26T12:00:28.572806+0000","last_became_peered":"2024-05-26T12:00:28.572806+0000","last_unstale":"2024-05-26T12:02:08.640473+0000","last_undegraded":"2024-05-26T12:02:08.640473+0000","last_fullsized":"2024-05-26T12:02:08.640473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:20:11.192942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640516+0000","last_change":"2024-05-26T12:01:08.470842+0000","last_active":"2024-05-26T12:02:08.640516+0000","last_peered":"2024-05-26T12:02:08.640516+0000","last_clean":"2024-05-26T12:02:08.640516+0000","last_became_active":"2024-05-26T12:01:08.470548+0000","last_became_peered":"2024-05-26T12:01:08.470548+0000","last_unstale":"2024-05-26T12:02:08.640516+0000","last_undegraded":"2024-05-26T12:02:08.640516+0000","last_fullsized":"2024-05-26T12:02:08.640516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:06:11.284570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538309+0000","last_change":"2024-05-26T12:00:28.672972+0000","last_active":"2024-05-26T12:02:08.538309+0000","last_peered":"2024-05-26T12:02:08.538309+0000","last_clean":"2024-05-26T12:02:08.538309+0000","last_became_active":"2024-05-26T12:00:28.672626+0000","last_became_peered":"2024-05-26T12:00:28.672626+0000","last_unstale":"2024-05-26T12:02:08.538309+0000","last_undegraded":"2024-05-26T12:02:08.538309+0000","last_fullsized":"2024-05-26T12:02:08.538309+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:34:31.449986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579574+0000","last_change":"2024-05-26T12:00:32.622325+0000","last_active":"2024-05-26T12:02:08.579574+0000","last_peered":"2024-05-26T12:02:08.579574+0000","last_clean":"2024-05-26T12:02:08.579574+0000","last_became_active":"2024-05-26T12:00:32.621970+0000","last_became_peered":"2024-05-26T12:00:32.621970+0000","last_unstale":"2024-05-26T12:02:08.579574+0000","last_undegraded":"2024-05-26T12:02:08.579574+0000","last_fullsized":"2024-05-26T12:02:08.579574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:32:36.252057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:30.684045+0000","last_change":"2024-05-26T12:00:28.580235+0000","last_active":"2024-05-26T12:02:30.684045+0000","last_peered":"2024-05-26T12:02:30.684045+0000","last_clean":"2024-05-26T12:02:30.684045+0000","last_became_active":"2024-05-26T12:00:28.579812+0000","last_became_peered":"2024-05-26T12:00:28.579812+0000","last_unstale":"2024-05-26T12:02:30.684045+0000","last_undegraded":"2024-05-26T12:02:30.684045+0000","last_fullsized":"2024-05-26T12:02:30.684045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:10:06.107316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579380+0000","last_change":"2024-05-26T12:00:32.625187+0000","last_active":"2024-05-26T12:02:08.579380+0000","last_peered":"2024-05-26T12:02:08.579380+0000","last_clean":"2024-05-26T12:02:08.579380+0000","last_became_active":"2024-05-26T12:00:32.624833+0000","last_became_peered":"2024-05-26T12:00:32.624833+0000","last_unstale":"2024-05-26T12:02:08.579380+0000","last_undegraded":"2024-05-26T12:02:08.579380+0000","last_fullsized":"2024-05-26T12:02:08.579380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:49:06.502017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582204+0000","last_change":"2024-05-26T12:01:07.950791+0000","last_active":"2024-05-26T12:02:08.582204+0000","last_peered":"2024-05-26T12:02:08.582204+0000","last_clean":"2024-05-26T12:02:08.582204+0000","last_became_active":"2024-05-26T12:01:07.950397+0000","last_became_peered":"2024-05-26T12:01:07.950397+0000","last_unstale":"2024-05-26T12:02:08.582204+0000","last_undegraded":"2024-05-26T12:02:08.582204+0000","last_fullsized":"2024-05-26T12:02:08.582204+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:20:15.193328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":135,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:29.659171+0000","last_change":"2024-05-26T12:00:28.671869+0000","last_active":"2024-05-26T12:02:29.659171+0000","last_peered":"2024-05-26T12:02:29.659171+0000","last_clean":"2024-05-26T12:02:29.659171+0000","last_became_active":"2024-05-26T12:00:28.671529+0000","last_became_peered":"2024-05-26T12:00:28.671529+0000","last_unstale":"2024-05-26T12:02:29.659171+0000","last_undegraded":"2024-05-26T12:02:29.659171+0000","last_fullsized":"2024-05-26T12:02:29.659171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:17.224354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":100,"num_read_kb":100,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536965+0000","last_change":"2024-05-26T12:00:32.621164+0000","last_active":"2024-05-26T12:02:08.536965+0000","last_peered":"2024-05-26T12:02:08.536965+0000","last_clean":"2024-05-26T12:02:08.536965+0000","last_became_active":"2024-05-26T12:00:32.618422+0000","last_became_peered":"2024-05-26T12:00:32.618422+0000","last_unstale":"2024-05-26T12:02:08.536965+0000","last_undegraded":"2024-05-26T12:02:08.536965+0000","last_fullsized":"2024-05-26T12:02:08.536965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:33:29.567862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621709+0000","last_change":"2024-05-26T12:00:28.579949+0000","last_active":"2024-05-26T12:02:08.621709+0000","last_peered":"2024-05-26T12:02:08.621709+0000","last_clean":"2024-05-26T12:02:08.621709+0000","last_became_active":"2024-05-26T12:00:28.579587+0000","last_became_peered":"2024-05-26T12:00:28.579587+0000","last_unstale":"2024-05-26T12:02:08.621709+0000","last_undegraded":"2024-05-26T12:02:08.621709+0000","last_fullsized":"2024-05-26T12:02:08.621709+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:26:18.491139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639976+0000","last_change":"2024-05-26T11:59:02.705869+0000","last_active":"2024-05-26T12:02:08.639976+0000","last_peered":"2024-05-26T12:02:08.639976+0000","last_clean":"2024-05-26T12:02:08.639976+0000","last_became_active":"2024-05-26T11:59:02.700918+0000","last_became_peered":"2024-05-26T11:59:02.700918+0000","last_unstale":"2024-05-26T12:02:08.639976+0000","last_undegraded":"2024-05-26T12:02:08.639976+0000","last_fullsized":"2024-05-26T12:02:08.639976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_clean_scrub_stamp":"2024-05-26T11:55:35.143931+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:01:24.735040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618717+0000","last_change":"2024-05-26T12:00:32.606843+0000","last_active":"2024-05-26T12:02:08.618717+0000","last_peered":"2024-05-26T12:02:08.618717+0000","last_clean":"2024-05-26T12:02:08.618717+0000","last_became_active":"2024-05-26T12:00:32.606552+0000","last_became_peered":"2024-05-26T12:00:32.606552+0000","last_unstale":"2024-05-26T12:02:08.618717+0000","last_undegraded":"2024-05-26T12:02:08.618717+0000","last_fullsized":"2024-05-26T12:02:08.618717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:43:03.113199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640311+0000","last_change":"2024-05-26T12:00:28.581819+0000","last_active":"2024-05-26T12:02:08.640311+0000","last_peered":"2024-05-26T12:02:08.640311+0000","last_clean":"2024-05-26T12:02:08.640311+0000","last_became_active":"2024-05-26T12:00:28.581430+0000","last_became_peered":"2024-05-26T12:00:28.581430+0000","last_unstale":"2024-05-26T12:02:08.640311+0000","last_undegraded":"2024-05-26T12:02:08.640311+0000","last_fullsized":"2024-05-26T12:02:08.640311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:13:42.556238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640535+0000","last_change":"2024-05-26T12:00:28.672396+0000","last_active":"2024-05-26T12:02:08.640535+0000","last_peered":"2024-05-26T12:02:08.640535+0000","last_clean":"2024-05-26T12:02:08.640535+0000","last_became_active":"2024-05-26T12:00:28.672045+0000","last_became_peered":"2024-05-26T12:00:28.672045+0000","last_unstale":"2024-05-26T12:02:08.640535+0000","last_undegraded":"2024-05-26T12:02:08.640535+0000","last_fullsized":"2024-05-26T12:02:08.640535+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:02:25.925210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618752+0000","last_change":"2024-05-26T12:00:32.606876+0000","last_active":"2024-05-26T12:02:08.618752+0000","last_peered":"2024-05-26T12:02:08.618752+0000","last_clean":"2024-05-26T12:02:08.618752+0000","last_became_active":"2024-05-26T12:00:32.606161+0000","last_became_peered":"2024-05-26T12:00:32.606161+0000","last_unstale":"2024-05-26T12:02:08.618752+0000","last_undegraded":"2024-05-26T12:02:08.618752+0000","last_fullsized":"2024-05-26T12:02:08.618752+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:47:02.451447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'15","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538445+0000","last_change":"2024-05-26T12:01:07.955816+0000","last_active":"2024-05-26T12:02:08.538445+0000","last_peered":"2024-05-26T12:02:08.538445+0000","last_clean":"2024-05-26T12:02:08.538445+0000","last_became_active":"2024-05-26T12:01:07.947443+0000","last_became_peered":"2024-05-26T12:01:07.947443+0000","last_unstale":"2024-05-26T12:02:08.538445+0000","last_undegraded":"2024-05-26T12:02:08.538445+0000","last_fullsized":"2024-05-26T12:02:08.538445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:30:00.005339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.642399+0000","last_change":"2024-05-26T12:02:08.642045+0000","last_active":"2024-05-26T12:02:08.642399+0000","last_peered":"2024-05-26T12:02:08.642399+0000","last_clean":"2024-05-26T12:02:08.642399+0000","last_became_active":"2024-05-26T12:02:08.641734+0000","last_became_peered":"2024-05-26T12:02:08.641734+0000","last_unstale":"2024-05-26T12:02:08.642399+0000","last_undegraded":"2024-05-26T12:02:08.642399+0000","last_fullsized":"2024-05-26T12:02:08.642399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:19:09.909218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.622031+0000","last_change":"2024-05-26T12:00:28.971347+0000","last_active":"2024-05-26T12:02:08.622031+0000","last_peered":"2024-05-26T12:02:08.622031+0000","last_clean":"2024-05-26T12:02:08.622031+0000","last_became_active":"2024-05-26T12:00:28.970911+0000","last_became_peered":"2024-05-26T12:00:28.970911+0000","last_unstale":"2024-05-26T12:02:08.622031+0000","last_undegraded":"2024-05-26T12:02:08.622031+0000","last_fullsized":"2024-05-26T12:02:08.622031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:47:45.306875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.834913+0000","last_change":"2024-05-26T12:00:32.614238+0000","last_active":"2024-05-26T12:02:08.834913+0000","last_peered":"2024-05-26T12:02:08.834913+0000","last_clean":"2024-05-26T12:02:08.834913+0000","last_became_active":"2024-05-26T12:00:32.607018+0000","last_became_peered":"2024-05-26T12:00:32.607018+0000","last_unstale":"2024-05-26T12:02:08.834913+0000","last_undegraded":"2024-05-26T12:02:08.834913+0000","last_fullsized":"2024-05-26T12:02:08.834913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:31:06.250615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538122+0000","last_change":"2024-05-26T12:00:28.671961+0000","last_active":"2024-05-26T12:02:08.538122+0000","last_peered":"2024-05-26T12:02:08.538122+0000","last_clean":"2024-05-26T12:02:08.538122+0000","last_became_active":"2024-05-26T12:00:28.671672+0000","last_became_peered":"2024-05-26T12:00:28.671672+0000","last_unstale":"2024-05-26T12:02:08.538122+0000","last_undegraded":"2024-05-26T12:02:08.538122+0000","last_fullsized":"2024-05-26T12:02:08.538122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:05.646044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639950+0000","last_change":"2024-05-26T12:00:32.611432+0000","last_active":"2024-05-26T12:02:08.639950+0000","last_peered":"2024-05-26T12:02:08.639950+0000","last_clean":"2024-05-26T12:02:08.639950+0000","last_became_active":"2024-05-26T12:00:32.610686+0000","last_became_peered":"2024-05-26T12:00:32.610686+0000","last_unstale":"2024-05-26T12:02:08.639950+0000","last_undegraded":"2024-05-26T12:02:08.639950+0000","last_fullsized":"2024-05-26T12:02:08.639950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:52:36.459552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536814+0000","last_change":"2024-05-26T12:00:28.576723+0000","last_active":"2024-05-26T12:02:08.536814+0000","last_peered":"2024-05-26T12:02:08.536814+0000","last_clean":"2024-05-26T12:02:08.536814+0000","last_became_active":"2024-05-26T12:00:28.576380+0000","last_became_peered":"2024-05-26T12:00:28.576380+0000","last_unstale":"2024-05-26T12:02:08.536814+0000","last_undegraded":"2024-05-26T12:02:08.536814+0000","last_fullsized":"2024-05-26T12:02:08.536814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:49:25.611830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640945+0000","last_change":"2024-05-26T12:00:32.621124+0000","last_active":"2024-05-26T12:02:08.640945+0000","last_peered":"2024-05-26T12:02:08.640945+0000","last_clean":"2024-05-26T12:02:08.640945+0000","last_became_active":"2024-05-26T12:00:32.620740+0000","last_became_peered":"2024-05-26T12:00:32.620740+0000","last_unstale":"2024-05-26T12:02:08.640945+0000","last_undegraded":"2024-05-26T12:02:08.640945+0000","last_fullsized":"2024-05-26T12:02:08.640945+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:15:16.123821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537934+0000","last_change":"2024-05-26T12:00:28.572222+0000","last_active":"2024-05-26T12:02:08.537934+0000","last_peered":"2024-05-26T12:02:08.537934+0000","last_clean":"2024-05-26T12:02:08.537934+0000","last_became_active":"2024-05-26T12:00:28.571459+0000","last_became_peered":"2024-05-26T12:00:28.571459+0000","last_unstale":"2024-05-26T12:02:08.537934+0000","last_undegraded":"2024-05-26T12:02:08.537934+0000","last_fullsized":"2024-05-26T12:02:08.537934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:49:56.234550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537165+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:02:08.537165+0000","last_peered":"2024-05-26T12:02:08.537165+0000","last_clean":"2024-05-26T12:02:08.537165+0000","last_became_active":"2024-05-26T12:00:32.617579+0000","last_became_peered":"2024-05-26T12:00:32.617579+0000","last_unstale":"2024-05-26T12:02:08.537165+0000","last_undegraded":"2024-05-26T12:02:08.537165+0000","last_fullsized":"2024-05-26T12:02:08.537165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:46:57.045155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640965+0000","last_change":"2024-05-26T12:00:28.584555+0000","last_active":"2024-05-26T12:02:08.640965+0000","last_peered":"2024-05-26T12:02:08.640965+0000","last_clean":"2024-05-26T12:02:08.640965+0000","last_became_active":"2024-05-26T12:00:28.583750+0000","last_became_peered":"2024-05-26T12:00:28.583750+0000","last_unstale":"2024-05-26T12:02:08.640965+0000","last_undegraded":"2024-05-26T12:02:08.640965+0000","last_fullsized":"2024-05-26T12:02:08.640965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:26:50.312369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639766+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:02:08.639766+0000","last_peered":"2024-05-26T12:02:08.639766+0000","last_clean":"2024-05-26T12:02:08.639766+0000","last_became_active":"2024-05-26T12:00:32.615758+0000","last_became_peered":"2024-05-26T12:00:32.615758+0000","last_unstale":"2024-05-26T12:02:08.639766+0000","last_undegraded":"2024-05-26T12:02:08.639766+0000","last_fullsized":"2024-05-26T12:02:08.639766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:17:52.776655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":134,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:27.664059+0000","last_change":"2024-05-26T12:00:28.971186+0000","last_active":"2024-05-26T12:02:27.664059+0000","last_peered":"2024-05-26T12:02:27.664059+0000","last_clean":"2024-05-26T12:02:27.664059+0000","last_became_active":"2024-05-26T12:00:28.970706+0000","last_became_peered":"2024-05-26T12:00:28.970706+0000","last_unstale":"2024-05-26T12:02:27.664059+0000","last_undegraded":"2024-05-26T12:02:27.664059+0000","last_fullsized":"2024-05-26T12:02:27.664059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:39:12.139262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":99,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639815+0000","last_change":"2024-05-26T12:00:32.623244+0000","last_active":"2024-05-26T12:02:08.639815+0000","last_peered":"2024-05-26T12:02:08.639815+0000","last_clean":"2024-05-26T12:02:08.639815+0000","last_became_active":"2024-05-26T12:00:32.622057+0000","last_became_peered":"2024-05-26T12:00:32.622057+0000","last_unstale":"2024-05-26T12:02:08.639815+0000","last_undegraded":"2024-05-26T12:02:08.639815+0000","last_fullsized":"2024-05-26T12:02:08.639815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:37:09.871626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582143+0000","last_change":"2024-05-26T12:00:28.576115+0000","last_active":"2024-05-26T12:02:08.582143+0000","last_peered":"2024-05-26T12:02:08.582143+0000","last_clean":"2024-05-26T12:02:08.582143+0000","last_became_active":"2024-05-26T12:00:28.575704+0000","last_became_peered":"2024-05-26T12:00:28.575704+0000","last_unstale":"2024-05-26T12:02:08.582143+0000","last_undegraded":"2024-05-26T12:02:08.582143+0000","last_fullsized":"2024-05-26T12:02:08.582143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:28:05.142390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640027+0000","last_change":"2024-05-26T12:00:32.611997+0000","last_active":"2024-05-26T12:02:08.640027+0000","last_peered":"2024-05-26T12:02:08.640027+0000","last_clean":"2024-05-26T12:02:08.640027+0000","last_became_active":"2024-05-26T12:00:32.611646+0000","last_became_peered":"2024-05-26T12:00:32.611646+0000","last_unstale":"2024-05-26T12:02:08.640027+0000","last_undegraded":"2024-05-26T12:02:08.640027+0000","last_fullsized":"2024-05-26T12:02:08.640027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:50:15.856048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619497+0000","last_change":"2024-05-26T12:00:28.576500+0000","last_active":"2024-05-26T12:02:08.619497+0000","last_peered":"2024-05-26T12:02:08.619497+0000","last_clean":"2024-05-26T12:02:08.619497+0000","last_became_active":"2024-05-26T12:00:28.576171+0000","last_became_peered":"2024-05-26T12:00:28.576171+0000","last_unstale":"2024-05-26T12:02:08.619497+0000","last_undegraded":"2024-05-26T12:02:08.619497+0000","last_fullsized":"2024-05-26T12:02:08.619497+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:44:54.786997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640451+0000","last_change":"2024-05-26T12:00:32.617550+0000","last_active":"2024-05-26T12:02:08.640451+0000","last_peered":"2024-05-26T12:02:08.640451+0000","last_clean":"2024-05-26T12:02:08.640451+0000","last_became_active":"2024-05-26T12:00:32.616821+0000","last_became_peered":"2024-05-26T12:00:32.616821+0000","last_unstale":"2024-05-26T12:02:08.640451+0000","last_undegraded":"2024-05-26T12:02:08.640451+0000","last_fullsized":"2024-05-26T12:02:08.640451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:49:59.956694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640212+0000","last_change":"2024-05-26T12:00:32.623314+0000","last_active":"2024-05-26T12:02:08.640212+0000","last_peered":"2024-05-26T12:02:08.640212+0000","last_clean":"2024-05-26T12:02:08.640212+0000","last_became_active":"2024-05-26T12:00:32.622187+0000","last_became_peered":"2024-05-26T12:00:32.622187+0000","last_unstale":"2024-05-26T12:02:08.640212+0000","last_undegraded":"2024-05-26T12:02:08.640212+0000","last_fullsized":"2024-05-26T12:02:08.640212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:19:08.026803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537778+0000","last_change":"2024-05-26T12:00:32.614814+0000","last_active":"2024-05-26T12:02:08.537778+0000","last_peered":"2024-05-26T12:02:08.537778+0000","last_clean":"2024-05-26T12:02:08.537778+0000","last_became_active":"2024-05-26T12:00:32.614439+0000","last_became_peered":"2024-05-26T12:00:32.614439+0000","last_unstale":"2024-05-26T12:02:08.537778+0000","last_undegraded":"2024-05-26T12:02:08.537778+0000","last_fullsized":"2024-05-26T12:02:08.537778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:56:10.817914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537746+0000","last_change":"2024-05-26T12:00:32.610170+0000","last_active":"2024-05-26T12:02:08.537746+0000","last_peered":"2024-05-26T12:02:08.537746+0000","last_clean":"2024-05-26T12:02:08.537746+0000","last_became_active":"2024-05-26T12:00:32.609783+0000","last_became_peered":"2024-05-26T12:00:32.609783+0000","last_unstale":"2024-05-26T12:02:08.537746+0000","last_undegraded":"2024-05-26T12:02:08.537746+0000","last_fullsized":"2024-05-26T12:02:08.537746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:33:52.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835448+0000","last_change":"2024-05-26T12:00:32.609819+0000","last_active":"2024-05-26T12:02:08.835448+0000","last_peered":"2024-05-26T12:02:08.835448+0000","last_clean":"2024-05-26T12:02:08.835448+0000","last_became_active":"2024-05-26T12:00:32.609016+0000","last_became_peered":"2024-05-26T12:00:32.609016+0000","last_unstale":"2024-05-26T12:02:08.835448+0000","last_undegraded":"2024-05-26T12:02:08.835448+0000","last_fullsized":"2024-05-26T12:02:08.835448+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:10:09.582603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619395+0000","last_change":"2024-05-26T12:00:32.608904+0000","last_active":"2024-05-26T12:02:08.619395+0000","last_peered":"2024-05-26T12:02:08.619395+0000","last_clean":"2024-05-26T12:02:08.619395+0000","last_became_active":"2024-05-26T12:00:32.608505+0000","last_became_peered":"2024-05-26T12:00:32.608505+0000","last_unstale":"2024-05-26T12:02:08.619395+0000","last_undegraded":"2024-05-26T12:02:08.619395+0000","last_fullsized":"2024-05-26T12:02:08.619395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:34:09.931046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640467+0000","last_change":"2024-05-26T12:00:32.618370+0000","last_active":"2024-05-26T12:02:08.640467+0000","last_peered":"2024-05-26T12:02:08.640467+0000","last_clean":"2024-05-26T12:02:08.640467+0000","last_became_active":"2024-05-26T12:00:32.618022+0000","last_became_peered":"2024-05-26T12:00:32.618022+0000","last_unstale":"2024-05-26T12:02:08.640467+0000","last_undegraded":"2024-05-26T12:02:08.640467+0000","last_fullsized":"2024-05-26T12:02:08.640467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:45:12.652460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537935+0000","last_change":"2024-05-26T12:00:32.612319+0000","last_active":"2024-05-26T12:02:08.537935+0000","last_peered":"2024-05-26T12:02:08.537935+0000","last_clean":"2024-05-26T12:02:08.537935+0000","last_became_active":"2024-05-26T12:00:32.611973+0000","last_became_peered":"2024-05-26T12:00:32.611973+0000","last_unstale":"2024-05-26T12:02:08.537935+0000","last_undegraded":"2024-05-26T12:02:08.537935+0000","last_fullsized":"2024-05-26T12:02:08.537935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:23:48.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536954+0000","last_change":"2024-05-26T12:00:32.613461+0000","last_active":"2024-05-26T12:02:08.536954+0000","last_peered":"2024-05-26T12:02:08.536954+0000","last_clean":"2024-05-26T12:02:08.536954+0000","last_became_active":"2024-05-26T12:00:32.613068+0000","last_became_peered":"2024-05-26T12:00:32.613068+0000","last_unstale":"2024-05-26T12:02:08.536954+0000","last_undegraded":"2024-05-26T12:02:08.536954+0000","last_fullsized":"2024-05-26T12:02:08.536954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:17:59.606664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581931+0000","last_change":"2024-05-26T12:00:32.615583+0000","last_active":"2024-05-26T12:02:08.581931+0000","last_peered":"2024-05-26T12:02:08.581931+0000","last_clean":"2024-05-26T12:02:08.581931+0000","last_became_active":"2024-05-26T12:00:32.611502+0000","last_became_peered":"2024-05-26T12:00:32.611502+0000","last_unstale":"2024-05-26T12:02:08.581931+0000","last_undegraded":"2024-05-26T12:02:08.581931+0000","last_fullsized":"2024-05-26T12:02:08.581931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:26:47.772010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536763+0000","last_change":"2024-05-26T12:00:32.612190+0000","last_active":"2024-05-26T12:02:08.536763+0000","last_peered":"2024-05-26T12:02:08.536763+0000","last_clean":"2024-05-26T12:02:08.536763+0000","last_became_active":"2024-05-26T12:00:32.611843+0000","last_became_peered":"2024-05-26T12:00:32.611843+0000","last_unstale":"2024-05-26T12:02:08.536763+0000","last_undegraded":"2024-05-26T12:02:08.536763+0000","last_fullsized":"2024-05-26T12:02:08.536763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:23:05.244741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538393+0000","last_change":"2024-05-26T12:00:32.620758+0000","last_active":"2024-05-26T12:02:08.538393+0000","last_peered":"2024-05-26T12:02:08.538393+0000","last_clean":"2024-05-26T12:02:08.538393+0000","last_became_active":"2024-05-26T12:00:32.620401+0000","last_became_peered":"2024-05-26T12:00:32.620401+0000","last_unstale":"2024-05-26T12:02:08.538393+0000","last_undegraded":"2024-05-26T12:02:08.538393+0000","last_fullsized":"2024-05-26T12:02:08.538393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:59:18.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619247+0000","last_change":"2024-05-26T12:00:32.615615+0000","last_active":"2024-05-26T12:02:08.619247+0000","last_peered":"2024-05-26T12:02:08.619247+0000","last_clean":"2024-05-26T12:02:08.619247+0000","last_became_active":"2024-05-26T12:00:32.615304+0000","last_became_peered":"2024-05-26T12:00:32.615304+0000","last_unstale":"2024-05-26T12:02:08.619247+0000","last_undegraded":"2024-05-26T12:02:08.619247+0000","last_fullsized":"2024-05-26T12:02:08.619247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:05:22.568910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640390+0000","last_change":"2024-05-26T12:00:32.627209+0000","last_active":"2024-05-26T12:02:08.640390+0000","last_peered":"2024-05-26T12:02:08.640390+0000","last_clean":"2024-05-26T12:02:08.640390+0000","last_became_active":"2024-05-26T12:00:32.626851+0000","last_became_peered":"2024-05-26T12:00:32.626851+0000","last_unstale":"2024-05-26T12:02:08.640390+0000","last_undegraded":"2024-05-26T12:02:08.640390+0000","last_fullsized":"2024-05-26T12:02:08.640390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:05:40.754565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213552,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66575,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1009,"num_read_kb":571,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":28173,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":199725,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678056,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":445991,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874285,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":1096692,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037811,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":445831,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234040,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":446209,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":1038727,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593795,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":445939,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574940,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":506614,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":933888,"data_stored":446887,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":446681,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149778,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":445373,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953549,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":505171,"data_compressed":16130,"data_compressed_allocated":24576,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790023,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":1037924,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60462,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T12:02:34.803 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph pg dump --format=json 2024-05-26T12:02:34.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:34 smithi186 ceph-mon[36412]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:35.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:34 smithi192 ceph-mon[36545]: pgmap v356: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:35.285 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:35.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:35 smithi156 ceph-mon[32025]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:35.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.662 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:35 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:35 smithi186 ceph-mon[36412]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:35 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:35 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:35 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:35.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:35 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:35 smithi192 ceph-mon[36545]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:35 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:35 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:35 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:36.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:35 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:36.762 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:36 smithi156 ceph-mon[32025]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:36.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:36 smithi186 ceph-mon[36412]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:37.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.003+0000 7feb97793700 1 -- 172.21.15.156:0/2172362010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102e00 msgr2=0x7feb9010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:37.007 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.003+0000 7feb97793700 1 --2- 172.21.15.156:0/2172362010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102e00 0x7feb9010f2f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7feb8800b0a0 tx=0x7feb8800b3b0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- 172.21.15.156:0/2172362010 shutdown_connections 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- 172.21.15.156:0/2172362010 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102e00 0x7feb9010f2f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- 172.21.15.156:0/2172362010 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90108520 0x7feb90108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- 172.21.15.156:0/2172362010 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90102440 0x7feb901028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- 172.21.15.156:0/2172362010 >> 172.21.15.156:0/2172362010 conn(0x7feb900793f0 msgr2=0x7feb90079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- 172.21.15.156:0/2172362010 shutdown_connections 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- 172.21.15.156:0/2172362010 wait complete. 2024-05-26T12:02:37.008 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 Processor -- start 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- start start 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90102e00 0x7feb9019b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90108520 0x7feb901a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7feb901126a0 con 0x7feb90102440 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7feb90112520 con 0x7feb90102e00 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.004+0000 7feb97793700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7feb90112820 con 0x7feb90108520 2024-05-26T12:02:37.009 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51776/0 (socket says 172.21.15.156:51776) 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 -- 172.21.15.156:0/2482963891 learned_addr learned my addr 172.21.15.156:0/2482963891 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb94d2e700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90102e00 0x7feb9019b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb95d30700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90108520 0x7feb901a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 -- 172.21.15.156:0/2482963891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90108520 msgr2=0x7feb901a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90108520 0x7feb901a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.010 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 -- 172.21.15.156:0/2482963891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90102e00 msgr2=0x7feb9019b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90102e00 0x7feb9019b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 -- 172.21.15.156:0/2482963891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb901a60c0 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb9552f700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feb8000d950 tx=0x7feb8000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb80016070 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb901a6290 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb80009df0 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8000f3c0 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.005+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb901a6820 con 0x7feb90102440 2024-05-26T12:02:37.011 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.006+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb9006a230 con 0x7feb90102440 2024-05-26T12:02:37.012 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.008+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 17) ==== 149704+0+0 (secure 0 0 0) 0x7feb80019070 con 0x7feb90102440 2024-05-26T12:02:37.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.009+0000 7feb867fc700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 0x7feb7c0b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:37.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.009+0000 7feb94d2e700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 0x7feb7c0b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:37.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.009+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7feb800e1fc0 con 0x7feb90102440 2024-05-26T12:02:37.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.009+0000 7feb94d2e700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 0x7feb7c0b4dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb900799d0 tx=0x7feb8c009190 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:37.013 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.012+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7feb800a9b50 con 0x7feb90102440 2024-05-26T12:02:37.072 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:36 smithi192 ceph-mon[36545]: pgmap v357: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:37.249 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.249+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7feb9006b7e0 con 0x7feb7c0b2910 2024-05-26T12:02:37.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.256+0000 7feb867fc700 1 -- 172.21.15.156:0/2482963891 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259024 (secure 0 0 0) 0x7feb9006b7e0 con 0x7feb7c0b2910 2024-05-26T12:02:37.259 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:37.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 msgr2=0x7feb7c0b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:37.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 0x7feb7c0b4dd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7feb900799d0 tx=0x7feb8c009190 comp rx=0 tx=0).stop 2024-05-26T12:02:37.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 msgr2=0x7feb9019ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:37.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7feb8000d950 tx=0x7feb8000dc60 comp rx=0 tx=0).stop 2024-05-26T12:02:37.261 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 shutdown_connections 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7feb7c0b2910 0x7feb7c0b4dd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7feb90108520 0x7feb901a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7feb90102440 0x7feb9019ab00 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 --2- 172.21.15.156:0/2482963891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb90102e00 0x7feb9019b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 >> 172.21.15.156:0/2482963891 conn(0x7feb900793f0 msgr2=0x7feb90100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 shutdown_connections 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:37.260+0000 7feb97793700 1 -- 172.21.15.156:0/2482963891 wait complete. 2024-05-26T12:02:37.262 INFO:teuthology.orchestra.run.smithi156.stderr:dumped all 2024-05-26T12:02:37.967 INFO:teuthology.orchestra.run.smithi156.stdout:{"pg_ready":true,"pg_map":{"version":358,"stamp":"2024-05-26T12:02:36.825477+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657177,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1130,"num_read_kb":660,"num_write":369,"num_write_kb":2286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2980013,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356240,"kb_used_data":11168,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585584,"statfs":{"total":1151940427776,"available":1151575638016,"internally_reserved":0,"allocated":11436032,"data_stored":7308039,"data_compressed":251796,"data_compressed_allocated":339968,"data_compressed_original":2187264,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":125,"num_read_kb":70,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002809"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640541+0000","last_change":"2024-05-26T12:00:32.620823+0000","last_active":"2024-05-26T12:02:08.640541+0000","last_peered":"2024-05-26T12:02:08.640541+0000","last_clean":"2024-05-26T12:02:08.640541+0000","last_became_active":"2024-05-26T12:00:32.619171+0000","last_became_peered":"2024-05-26T12:00:32.619171+0000","last_unstale":"2024-05-26T12:02:08.640541+0000","last_undegraded":"2024-05-26T12:02:08.640541+0000","last_fullsized":"2024-05-26T12:02:08.640541+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:57:59.459768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835075+0000","last_change":"2024-05-26T12:00:32.618888+0000","last_active":"2024-05-26T12:02:08.835075+0000","last_peered":"2024-05-26T12:02:08.835075+0000","last_clean":"2024-05-26T12:02:08.835075+0000","last_became_active":"2024-05-26T12:00:32.618235+0000","last_became_peered":"2024-05-26T12:00:32.618235+0000","last_unstale":"2024-05-26T12:02:08.835075+0000","last_undegraded":"2024-05-26T12:02:08.835075+0000","last_fullsized":"2024-05-26T12:02:08.835075+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:56:08.852594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.834949+0000","last_change":"2024-05-26T12:00:32.613694+0000","last_active":"2024-05-26T12:02:08.834949+0000","last_peered":"2024-05-26T12:02:08.834949+0000","last_clean":"2024-05-26T12:02:08.834949+0000","last_became_active":"2024-05-26T12:00:32.613389+0000","last_became_peered":"2024-05-26T12:00:32.613389+0000","last_unstale":"2024-05-26T12:02:08.834949+0000","last_undegraded":"2024-05-26T12:02:08.834949+0000","last_fullsized":"2024-05-26T12:02:08.834949+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:04.850384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621312+0000","last_change":"2024-05-26T12:00:32.620149+0000","last_active":"2024-05-26T12:02:08.621312+0000","last_peered":"2024-05-26T12:02:08.621312+0000","last_clean":"2024-05-26T12:02:08.621312+0000","last_became_active":"2024-05-26T12:00:32.614534+0000","last_became_peered":"2024-05-26T12:00:32.614534+0000","last_unstale":"2024-05-26T12:02:08.621312+0000","last_undegraded":"2024-05-26T12:02:08.621312+0000","last_fullsized":"2024-05-26T12:02:08.621312+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:24:41.144826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639969+0000","last_change":"2024-05-26T12:00:32.605841+0000","last_active":"2024-05-26T12:02:08.639969+0000","last_peered":"2024-05-26T12:02:08.639969+0000","last_clean":"2024-05-26T12:02:08.639969+0000","last_became_active":"2024-05-26T12:00:32.605438+0000","last_became_peered":"2024-05-26T12:00:32.605438+0000","last_unstale":"2024-05-26T12:02:08.639969+0000","last_undegraded":"2024-05-26T12:02:08.639969+0000","last_fullsized":"2024-05-26T12:02:08.639969+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:05:01.341398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579377+0000","last_change":"2024-05-26T12:00:32.625270+0000","last_active":"2024-05-26T12:02:08.579377+0000","last_peered":"2024-05-26T12:02:08.579377+0000","last_clean":"2024-05-26T12:02:08.579377+0000","last_became_active":"2024-05-26T12:00:32.624892+0000","last_became_peered":"2024-05-26T12:00:32.624892+0000","last_unstale":"2024-05-26T12:02:08.579377+0000","last_undegraded":"2024-05-26T12:02:08.579377+0000","last_fullsized":"2024-05-26T12:02:08.579377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:44.581793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639797+0000","last_change":"2024-05-26T12:00:32.613286+0000","last_active":"2024-05-26T12:02:08.639797+0000","last_peered":"2024-05-26T12:02:08.639797+0000","last_clean":"2024-05-26T12:02:08.639797+0000","last_became_active":"2024-05-26T12:00:32.612896+0000","last_became_peered":"2024-05-26T12:00:32.612896+0000","last_unstale":"2024-05-26T12:02:08.639797+0000","last_undegraded":"2024-05-26T12:02:08.639797+0000","last_fullsized":"2024-05-26T12:02:08.639797+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:34:29.796529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536725+0000","last_change":"2024-05-26T12:00:32.620975+0000","last_active":"2024-05-26T12:02:08.536725+0000","last_peered":"2024-05-26T12:02:08.536725+0000","last_clean":"2024-05-26T12:02:08.536725+0000","last_became_active":"2024-05-26T12:00:32.618576+0000","last_became_peered":"2024-05-26T12:00:32.618576+0000","last_unstale":"2024-05-26T12:02:08.536725+0000","last_undegraded":"2024-05-26T12:02:08.536725+0000","last_fullsized":"2024-05-26T12:02:08.536725+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:51:04.187910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538637+0000","last_change":"2024-05-26T12:00:32.618617+0000","last_active":"2024-05-26T12:02:08.538637+0000","last_peered":"2024-05-26T12:02:08.538637+0000","last_clean":"2024-05-26T12:02:08.538637+0000","last_became_active":"2024-05-26T12:00:32.617354+0000","last_became_peered":"2024-05-26T12:00:32.617354+0000","last_unstale":"2024-05-26T12:02:08.538637+0000","last_undegraded":"2024-05-26T12:02:08.538637+0000","last_fullsized":"2024-05-26T12:02:08.538637+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:04:22.469952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579758+0000","last_change":"2024-05-26T12:00:32.618169+0000","last_active":"2024-05-26T12:02:08.579758+0000","last_peered":"2024-05-26T12:02:08.579758+0000","last_clean":"2024-05-26T12:02:08.579758+0000","last_became_active":"2024-05-26T12:00:32.617709+0000","last_became_peered":"2024-05-26T12:00:32.617709+0000","last_unstale":"2024-05-26T12:02:08.579758+0000","last_undegraded":"2024-05-26T12:02:08.579758+0000","last_fullsized":"2024-05-26T12:02:08.579758+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:21.801512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639958+0000","last_change":"2024-05-26T12:00:32.623233+0000","last_active":"2024-05-26T12:02:08.639958+0000","last_peered":"2024-05-26T12:02:08.639958+0000","last_clean":"2024-05-26T12:02:08.639958+0000","last_became_active":"2024-05-26T12:00:32.622483+0000","last_became_peered":"2024-05-26T12:00:32.622483+0000","last_unstale":"2024-05-26T12:02:08.639958+0000","last_undegraded":"2024-05-26T12:02:08.639958+0000","last_fullsized":"2024-05-26T12:02:08.639958+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:14.784275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621365+0000","last_change":"2024-05-26T12:00:32.614575+0000","last_active":"2024-05-26T12:02:08.621365+0000","last_peered":"2024-05-26T12:02:08.621365+0000","last_clean":"2024-05-26T12:02:08.621365+0000","last_became_active":"2024-05-26T12:00:32.614020+0000","last_became_peered":"2024-05-26T12:00:32.614020+0000","last_unstale":"2024-05-26T12:02:08.621365+0000","last_undegraded":"2024-05-26T12:02:08.621365+0000","last_fullsized":"2024-05-26T12:02:08.621365+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:33:05.747419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618727+0000","last_change":"2024-05-26T12:00:32.616624+0000","last_active":"2024-05-26T12:02:08.618727+0000","last_peered":"2024-05-26T12:02:08.618727+0000","last_clean":"2024-05-26T12:02:08.618727+0000","last_became_active":"2024-05-26T12:00:32.616280+0000","last_became_peered":"2024-05-26T12:00:32.616280+0000","last_unstale":"2024-05-26T12:02:08.618727+0000","last_undegraded":"2024-05-26T12:02:08.618727+0000","last_fullsized":"2024-05-26T12:02:08.618727+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:45:02.466249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835014+0000","last_change":"2024-05-26T12:00:32.617058+0000","last_active":"2024-05-26T12:02:08.835014+0000","last_peered":"2024-05-26T12:02:08.835014+0000","last_clean":"2024-05-26T12:02:08.835014+0000","last_became_active":"2024-05-26T12:00:32.614651+0000","last_became_peered":"2024-05-26T12:00:32.614651+0000","last_unstale":"2024-05-26T12:02:08.835014+0000","last_undegraded":"2024-05-26T12:02:08.835014+0000","last_fullsized":"2024-05-26T12:02:08.835014+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:38:50.234745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835005+0000","last_change":"2024-05-26T12:01:07.950685+0000","last_active":"2024-05-26T12:02:08.835005+0000","last_peered":"2024-05-26T12:02:08.835005+0000","last_clean":"2024-05-26T12:02:08.835005+0000","last_became_active":"2024-05-26T12:01:07.950383+0000","last_became_peered":"2024-05-26T12:01:07.950383+0000","last_unstale":"2024-05-26T12:02:08.835005+0000","last_undegraded":"2024-05-26T12:02:08.835005+0000","last_fullsized":"2024-05-26T12:02:08.835005+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:02:42.832274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,2],"acting":[0,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538100+0000","last_change":"2024-05-26T12:00:32.618850+0000","last_active":"2024-05-26T12:02:08.538100+0000","last_peered":"2024-05-26T12:02:08.538100+0000","last_clean":"2024-05-26T12:02:08.538100+0000","last_became_active":"2024-05-26T12:00:32.618494+0000","last_became_peered":"2024-05-26T12:00:32.618494+0000","last_unstale":"2024-05-26T12:02:08.538100+0000","last_undegraded":"2024-05-26T12:02:08.538100+0000","last_fullsized":"2024-05-26T12:02:08.538100+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:23:43.555380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640263+0000","last_change":"2024-05-26T12:00:32.621515+0000","last_active":"2024-05-26T12:02:08.640263+0000","last_peered":"2024-05-26T12:02:08.640263+0000","last_clean":"2024-05-26T12:02:08.640263+0000","last_became_active":"2024-05-26T12:00:32.621091+0000","last_became_peered":"2024-05-26T12:00:32.621091+0000","last_unstale":"2024-05-26T12:02:08.640263+0000","last_undegraded":"2024-05-26T12:02:08.640263+0000","last_fullsized":"2024-05-26T12:02:08.640263+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:32:07.614386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538389+0000","last_change":"2024-05-26T12:00:32.612371+0000","last_active":"2024-05-26T12:02:08.538389+0000","last_peered":"2024-05-26T12:02:08.538389+0000","last_clean":"2024-05-26T12:02:08.538389+0000","last_became_active":"2024-05-26T12:00:32.612082+0000","last_became_peered":"2024-05-26T12:00:32.612082+0000","last_unstale":"2024-05-26T12:02:08.538389+0000","last_undegraded":"2024-05-26T12:02:08.538389+0000","last_fullsized":"2024-05-26T12:02:08.538389+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:25:15.850251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537943+0000","last_change":"2024-05-26T12:00:32.611421+0000","last_active":"2024-05-26T12:02:08.537943+0000","last_peered":"2024-05-26T12:02:08.537943+0000","last_clean":"2024-05-26T12:02:08.537943+0000","last_became_active":"2024-05-26T12:00:32.609716+0000","last_became_peered":"2024-05-26T12:00:32.609716+0000","last_unstale":"2024-05-26T12:02:08.537943+0000","last_undegraded":"2024-05-26T12:02:08.537943+0000","last_fullsized":"2024-05-26T12:02:08.537943+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:55.197945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"79'13","reported_seq":147,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:33.687330+0000","last_change":"2024-05-26T12:00:28.578034+0000","last_active":"2024-05-26T12:02:33.687330+0000","last_peered":"2024-05-26T12:02:33.687330+0000","last_clean":"2024-05-26T12:02:33.687330+0000","last_became_active":"2024-05-26T12:00:28.577682+0000","last_became_peered":"2024-05-26T12:00:28.577682+0000","last_unstale":"2024-05-26T12:02:33.687330+0000","last_undegraded":"2024-05-26T12:02:33.687330+0000","last_fullsized":"2024-05-26T12:02:33.687330+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:27:11.135886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":103,"num_read_kb":103,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537109+0000","last_change":"2024-05-26T12:00:32.609857+0000","last_active":"2024-05-26T12:02:08.537109+0000","last_peered":"2024-05-26T12:02:08.537109+0000","last_clean":"2024-05-26T12:02:08.537109+0000","last_became_active":"2024-05-26T12:00:32.609380+0000","last_became_peered":"2024-05-26T12:00:32.609380+0000","last_unstale":"2024-05-26T12:02:08.537109+0000","last_undegraded":"2024-05-26T12:02:08.537109+0000","last_fullsized":"2024-05-26T12:02:08.537109+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:39:04.869179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"79'2","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537252+0000","last_change":"2024-05-26T12:00:28.671940+0000","last_active":"2024-05-26T12:02:08.537252+0000","last_peered":"2024-05-26T12:02:08.537252+0000","last_clean":"2024-05-26T12:02:08.537252+0000","last_became_active":"2024-05-26T12:00:28.671602+0000","last_became_peered":"2024-05-26T12:00:28.671602+0000","last_unstale":"2024-05-26T12:02:08.537252+0000","last_undegraded":"2024-05-26T12:02:08.537252+0000","last_fullsized":"2024-05-26T12:02:08.537252+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:38:52.069136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835015+0000","last_change":"2024-05-26T12:00:32.613900+0000","last_active":"2024-05-26T12:02:08.835015+0000","last_peered":"2024-05-26T12:02:08.835015+0000","last_clean":"2024-05-26T12:02:08.835015+0000","last_became_active":"2024-05-26T12:00:32.610264+0000","last_became_peered":"2024-05-26T12:00:32.610264+0000","last_unstale":"2024-05-26T12:02:08.835015+0000","last_undegraded":"2024-05-26T12:02:08.835015+0000","last_fullsized":"2024-05-26T12:02:08.835015+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:28:58.712711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"79'12","reported_seq":72,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640747+0000","last_change":"2024-05-26T12:01:08.483676+0000","last_active":"2024-05-26T12:02:08.640747+0000","last_peered":"2024-05-26T12:02:08.640747+0000","last_clean":"2024-05-26T12:02:08.640747+0000","last_became_active":"2024-05-26T12:01:08.471437+0000","last_became_peered":"2024-05-26T12:01:08.471437+0000","last_unstale":"2024-05-26T12:02:08.640747+0000","last_undegraded":"2024-05-26T12:02:08.640747+0000","last_fullsized":"2024-05-26T12:02:08.640747+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:43:38.663368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,6],"acting":[8,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538279+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:02:08.538279+0000","last_peered":"2024-05-26T12:02:08.538279+0000","last_clean":"2024-05-26T12:02:08.538279+0000","last_became_active":"2024-05-26T12:00:32.617702+0000","last_became_peered":"2024-05-26T12:00:32.617702+0000","last_unstale":"2024-05-26T12:02:08.538279+0000","last_undegraded":"2024-05-26T12:02:08.538279+0000","last_fullsized":"2024-05-26T12:02:08.538279+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:25:06.527677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"79'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640790+0000","last_change":"2024-05-26T12:00:28.575648+0000","last_active":"2024-05-26T12:02:08.640790+0000","last_peered":"2024-05-26T12:02:08.640790+0000","last_clean":"2024-05-26T12:02:08.640790+0000","last_became_active":"2024-05-26T12:00:28.575170+0000","last_became_peered":"2024-05-26T12:00:28.575170+0000","last_unstale":"2024-05-26T12:02:08.640790+0000","last_undegraded":"2024-05-26T12:02:08.640790+0000","last_fullsized":"2024-05-26T12:02:08.640790+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:09:29.213605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582175+0000","last_change":"2024-05-26T12:00:32.617825+0000","last_active":"2024-05-26T12:02:08.582175+0000","last_peered":"2024-05-26T12:02:08.582175+0000","last_clean":"2024-05-26T12:02:08.582175+0000","last_became_active":"2024-05-26T12:00:32.617517+0000","last_became_peered":"2024-05-26T12:00:32.617517+0000","last_unstale":"2024-05-26T12:02:08.582175+0000","last_undegraded":"2024-05-26T12:02:08.582175+0000","last_fullsized":"2024-05-26T12:02:08.582175+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:36:33.174637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"79'5","reported_seq":139,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:32.650523+0000","last_change":"2024-05-26T12:00:28.584639+0000","last_active":"2024-05-26T12:02:32.650523+0000","last_peered":"2024-05-26T12:02:32.650523+0000","last_clean":"2024-05-26T12:02:32.650523+0000","last_became_active":"2024-05-26T12:00:28.583063+0000","last_became_peered":"2024-05-26T12:00:28.583063+0000","last_unstale":"2024-05-26T12:02:32.650523+0000","last_undegraded":"2024-05-26T12:02:32.650523+0000","last_fullsized":"2024-05-26T12:02:32.650523+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:01:25.804547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":104,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639966+0000","last_change":"2024-05-26T12:00:32.609117+0000","last_active":"2024-05-26T12:02:08.639966+0000","last_peered":"2024-05-26T12:02:08.639966+0000","last_clean":"2024-05-26T12:02:08.639966+0000","last_became_active":"2024-05-26T12:00:32.608770+0000","last_became_peered":"2024-05-26T12:00:32.608770+0000","last_unstale":"2024-05-26T12:02:08.639966+0000","last_undegraded":"2024-05-26T12:02:08.639966+0000","last_fullsized":"2024-05-26T12:02:08.639966+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:19:57.422101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"79'7","reported_seq":51,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579332+0000","last_change":"2024-05-26T12:01:07.954880+0000","last_active":"2024-05-26T12:02:08.579332+0000","last_peered":"2024-05-26T12:02:08.579332+0000","last_clean":"2024-05-26T12:02:08.579332+0000","last_became_active":"2024-05-26T12:01:07.949506+0000","last_became_peered":"2024-05-26T12:01:07.949506+0000","last_unstale":"2024-05-26T12:02:08.579332+0000","last_undegraded":"2024-05-26T12:02:08.579332+0000","last_fullsized":"2024-05-26T12:02:08.579332+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:40:37.041619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582377+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:02:08.582377+0000","last_peered":"2024-05-26T12:02:08.582377+0000","last_clean":"2024-05-26T12:02:08.582377+0000","last_became_active":"2024-05-26T12:00:32.615691+0000","last_became_peered":"2024-05-26T12:00:32.615691+0000","last_unstale":"2024-05-26T12:02:08.582377+0000","last_undegraded":"2024-05-26T12:02:08.582377+0000","last_fullsized":"2024-05-26T12:02:08.582377+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:58:26.245149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"79'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619059+0000","last_change":"2024-05-26T12:00:28.571893+0000","last_active":"2024-05-26T12:02:08.619059+0000","last_peered":"2024-05-26T12:02:08.619059+0000","last_clean":"2024-05-26T12:02:08.619059+0000","last_became_active":"2024-05-26T12:00:28.571502+0000","last_became_peered":"2024-05-26T12:00:28.571502+0000","last_unstale":"2024-05-26T12:02:08.619059+0000","last_undegraded":"2024-05-26T12:02:08.619059+0000","last_fullsized":"2024-05-26T12:02:08.619059+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:08:01.040087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536788+0000","last_change":"2024-05-26T12:00:32.616575+0000","last_active":"2024-05-26T12:02:08.536788+0000","last_peered":"2024-05-26T12:02:08.536788+0000","last_clean":"2024-05-26T12:02:08.536788+0000","last_became_active":"2024-05-26T12:00:32.616261+0000","last_became_peered":"2024-05-26T12:00:32.616261+0000","last_unstale":"2024-05-26T12:02:08.536788+0000","last_undegraded":"2024-05-26T12:02:08.536788+0000","last_fullsized":"2024-05-26T12:02:08.536788+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:06:50.012838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581855+0000","last_change":"2024-05-26T12:00:28.584129+0000","last_active":"2024-05-26T12:02:08.581855+0000","last_peered":"2024-05-26T12:02:08.581855+0000","last_clean":"2024-05-26T12:02:08.581855+0000","last_became_active":"2024-05-26T12:00:28.583046+0000","last_became_peered":"2024-05-26T12:00:28.583046+0000","last_unstale":"2024-05-26T12:02:08.581855+0000","last_undegraded":"2024-05-26T12:02:08.581855+0000","last_fullsized":"2024-05-26T12:02:08.581855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:24:47.635696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538138+0000","last_change":"2024-05-26T12:00:32.610148+0000","last_active":"2024-05-26T12:02:08.538138+0000","last_peered":"2024-05-26T12:02:08.538138+0000","last_clean":"2024-05-26T12:02:08.538138+0000","last_became_active":"2024-05-26T12:00:32.609849+0000","last_became_peered":"2024-05-26T12:00:32.609849+0000","last_unstale":"2024-05-26T12:02:08.538138+0000","last_undegraded":"2024-05-26T12:02:08.538138+0000","last_fullsized":"2024-05-26T12:02:08.538138+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:37:08.938164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"79'4","reported_seq":240,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:34.652371+0000","last_change":"2024-05-26T12:00:28.578831+0000","last_active":"2024-05-26T12:02:34.652371+0000","last_peered":"2024-05-26T12:02:34.652371+0000","last_clean":"2024-05-26T12:02:34.652371+0000","last_became_active":"2024-05-26T12:00:28.578484+0000","last_became_peered":"2024-05-26T12:00:28.578484+0000","last_unstale":"2024-05-26T12:02:34.652371+0000","last_undegraded":"2024-05-26T12:02:34.652371+0000","last_fullsized":"2024-05-26T12:02:34.652371+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:11:07.127948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":206,"num_read_kb":206,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538626+0000","last_change":"2024-05-26T12:00:32.618151+0000","last_active":"2024-05-26T12:02:08.538626+0000","last_peered":"2024-05-26T12:02:08.538626+0000","last_clean":"2024-05-26T12:02:08.538626+0000","last_became_active":"2024-05-26T12:00:32.616835+0000","last_became_peered":"2024-05-26T12:00:32.616835+0000","last_unstale":"2024-05-26T12:02:08.538626+0000","last_undegraded":"2024-05-26T12:02:08.538626+0000","last_fullsized":"2024-05-26T12:02:08.538626+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:17:57.682573+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"79'4","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640171+0000","last_change":"2024-05-26T12:00:28.579624+0000","last_active":"2024-05-26T12:02:08.640171+0000","last_peered":"2024-05-26T12:02:08.640171+0000","last_clean":"2024-05-26T12:02:08.640171+0000","last_became_active":"2024-05-26T12:00:28.578691+0000","last_became_peered":"2024-05-26T12:00:28.578691+0000","last_unstale":"2024-05-26T12:02:08.640171+0000","last_undegraded":"2024-05-26T12:02:08.640171+0000","last_fullsized":"2024-05-26T12:02:08.640171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:38:12.319875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835233+0000","last_change":"2024-05-26T12:00:32.610573+0000","last_active":"2024-05-26T12:02:08.835233+0000","last_peered":"2024-05-26T12:02:08.835233+0000","last_clean":"2024-05-26T12:02:08.835233+0000","last_became_active":"2024-05-26T12:00:32.609824+0000","last_became_peered":"2024-05-26T12:00:32.609824+0000","last_unstale":"2024-05-26T12:02:08.835233+0000","last_undegraded":"2024-05-26T12:02:08.835233+0000","last_fullsized":"2024-05-26T12:02:08.835233+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:26:56.631874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"79'5","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582284+0000","last_change":"2024-05-26T12:00:28.672441+0000","last_active":"2024-05-26T12:02:08.582284+0000","last_peered":"2024-05-26T12:02:08.582284+0000","last_clean":"2024-05-26T12:02:08.582284+0000","last_became_active":"2024-05-26T12:00:28.672039+0000","last_became_peered":"2024-05-26T12:00:28.672039+0000","last_unstale":"2024-05-26T12:02:08.582284+0000","last_undegraded":"2024-05-26T12:02:08.582284+0000","last_fullsized":"2024-05-26T12:02:08.582284+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:14:09.757458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538369+0000","last_change":"2024-05-26T12:00:32.612289+0000","last_active":"2024-05-26T12:02:08.538369+0000","last_peered":"2024-05-26T12:02:08.538369+0000","last_clean":"2024-05-26T12:02:08.538369+0000","last_became_active":"2024-05-26T12:00:32.611930+0000","last_became_peered":"2024-05-26T12:00:32.611930+0000","last_unstale":"2024-05-26T12:02:08.538369+0000","last_undegraded":"2024-05-26T12:02:08.538369+0000","last_fullsized":"2024-05-26T12:02:08.538369+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:05:55.351242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"81'31","reported_seq":294,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:34.623931+0000","last_change":"2024-05-26T12:01:08.493089+0000","last_active":"2024-05-26T12:02:34.623931+0000","last_peered":"2024-05-26T12:02:34.623931+0000","last_clean":"2024-05-26T12:02:34.623931+0000","last_became_active":"2024-05-26T12:01:08.471966+0000","last_became_peered":"2024-05-26T12:01:08.471966+0000","last_unstale":"2024-05-26T12:02:34.623931+0000","last_undegraded":"2024-05-26T12:02:34.623931+0000","last_fullsized":"2024-05-26T12:02:34.623931+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:12:51.413474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59762,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":206,"num_read_kb":155,"num_write":33,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":28083,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640426+0000","last_change":"2024-05-26T12:00:32.614557+0000","last_active":"2024-05-26T12:02:08.640426+0000","last_peered":"2024-05-26T12:02:08.640426+0000","last_clean":"2024-05-26T12:02:08.640426+0000","last_became_active":"2024-05-26T12:00:32.613806+0000","last_became_peered":"2024-05-26T12:00:32.613806+0000","last_unstale":"2024-05-26T12:02:08.640426+0000","last_undegraded":"2024-05-26T12:02:08.640426+0000","last_fullsized":"2024-05-26T12:02:08.640426+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:11:11.279479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"79'9","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538147+0000","last_change":"2024-05-26T12:00:28.577551+0000","last_active":"2024-05-26T12:02:08.538147+0000","last_peered":"2024-05-26T12:02:08.538147+0000","last_clean":"2024-05-26T12:02:08.538147+0000","last_became_active":"2024-05-26T12:00:28.577203+0000","last_became_peered":"2024-05-26T12:00:28.577203+0000","last_unstale":"2024-05-26T12:02:08.538147+0000","last_undegraded":"2024-05-26T12:02:08.538147+0000","last_fullsized":"2024-05-26T12:02:08.538147+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:22:54.906432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618745+0000","last_change":"2024-05-26T12:00:32.612029+0000","last_active":"2024-05-26T12:02:08.618745+0000","last_peered":"2024-05-26T12:02:08.618745+0000","last_clean":"2024-05-26T12:02:08.618745+0000","last_became_active":"2024-05-26T12:00:32.611660+0000","last_became_peered":"2024-05-26T12:00:32.611660+0000","last_unstale":"2024-05-26T12:02:08.618745+0000","last_undegraded":"2024-05-26T12:02:08.618745+0000","last_fullsized":"2024-05-26T12:02:08.618745+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:15:27.278693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"79'5","reported_seq":23,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640375+0000","last_change":"2024-05-26T12:01:07.960293+0000","last_active":"2024-05-26T12:02:08.640375+0000","last_peered":"2024-05-26T12:02:08.640375+0000","last_clean":"2024-05-26T12:02:08.640375+0000","last_became_active":"2024-05-26T12:01:07.947178+0000","last_became_peered":"2024-05-26T12:01:07.947178+0000","last_unstale":"2024-05-26T12:02:08.640375+0000","last_undegraded":"2024-05-26T12:02:08.640375+0000","last_fullsized":"2024-05-26T12:02:08.640375+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:28:23.807776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":90,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,7],"acting":[11,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640288+0000","last_change":"2024-05-26T12:01:07.948672+0000","last_active":"2024-05-26T12:02:08.640288+0000","last_peered":"2024-05-26T12:02:08.640288+0000","last_clean":"2024-05-26T12:02:08.640288+0000","last_became_active":"2024-05-26T12:01:07.948285+0000","last_became_peered":"2024-05-26T12:01:07.948285+0000","last_unstale":"2024-05-26T12:02:08.640288+0000","last_undegraded":"2024-05-26T12:02:08.640288+0000","last_fullsized":"2024-05-26T12:02:08.640288+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:49:39.764352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"79'5","reported_seq":140,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:33.667627+0000","last_change":"2024-05-26T12:00:28.578507+0000","last_active":"2024-05-26T12:02:33.667627+0000","last_peered":"2024-05-26T12:02:33.667627+0000","last_clean":"2024-05-26T12:02:33.667627+0000","last_became_active":"2024-05-26T12:00:28.578131+0000","last_became_peered":"2024-05-26T12:00:28.578131+0000","last_unstale":"2024-05-26T12:02:33.667627+0000","last_undegraded":"2024-05-26T12:02:33.667627+0000","last_fullsized":"2024-05-26T12:02:33.667627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:14:52.015296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":106,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639749+0000","last_change":"2024-05-26T12:00:32.618061+0000","last_active":"2024-05-26T12:02:08.639749+0000","last_peered":"2024-05-26T12:02:08.639749+0000","last_clean":"2024-05-26T12:02:08.639749+0000","last_became_active":"2024-05-26T12:00:32.617320+0000","last_became_peered":"2024-05-26T12:00:32.617320+0000","last_unstale":"2024-05-26T12:02:08.639749+0000","last_undegraded":"2024-05-26T12:02:08.639749+0000","last_fullsized":"2024-05-26T12:02:08.639749+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:57:33.972120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"79'17","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640264+0000","last_change":"2024-05-26T12:00:28.672890+0000","last_active":"2024-05-26T12:02:08.640264+0000","last_peered":"2024-05-26T12:02:08.640264+0000","last_clean":"2024-05-26T12:02:08.640264+0000","last_became_active":"2024-05-26T12:00:28.672523+0000","last_became_peered":"2024-05-26T12:00:28.672523+0000","last_unstale":"2024-05-26T12:02:08.640264+0000","last_undegraded":"2024-05-26T12:02:08.640264+0000","last_fullsized":"2024-05-26T12:02:08.640264+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:46:00.281960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581992+0000","last_change":"2024-05-26T12:00:32.618020+0000","last_active":"2024-05-26T12:02:08.581992+0000","last_peered":"2024-05-26T12:02:08.581992+0000","last_clean":"2024-05-26T12:02:08.581992+0000","last_became_active":"2024-05-26T12:00:32.617147+0000","last_became_peered":"2024-05-26T12:00:32.617147+0000","last_unstale":"2024-05-26T12:02:08.581992+0000","last_undegraded":"2024-05-26T12:02:08.581992+0000","last_fullsized":"2024-05-26T12:02:08.581992+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:54:06.365456+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"79'2","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640473+0000","last_change":"2024-05-26T12:00:28.573265+0000","last_active":"2024-05-26T12:02:08.640473+0000","last_peered":"2024-05-26T12:02:08.640473+0000","last_clean":"2024-05-26T12:02:08.640473+0000","last_became_active":"2024-05-26T12:00:28.572806+0000","last_became_peered":"2024-05-26T12:00:28.572806+0000","last_unstale":"2024-05-26T12:02:08.640473+0000","last_undegraded":"2024-05-26T12:02:08.640473+0000","last_fullsized":"2024-05-26T12:02:08.640473+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:20:11.192942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640516+0000","last_change":"2024-05-26T12:01:08.470842+0000","last_active":"2024-05-26T12:02:08.640516+0000","last_peered":"2024-05-26T12:02:08.640516+0000","last_clean":"2024-05-26T12:02:08.640516+0000","last_became_active":"2024-05-26T12:01:08.470548+0000","last_became_peered":"2024-05-26T12:01:08.470548+0000","last_unstale":"2024-05-26T12:02:08.640516+0000","last_undegraded":"2024-05-26T12:02:08.640516+0000","last_fullsized":"2024-05-26T12:02:08.640516+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:06:11.284570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"79'1","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538361+0000","last_change":"2024-05-26T12:00:28.672972+0000","last_active":"2024-05-26T12:02:08.538361+0000","last_peered":"2024-05-26T12:02:08.538361+0000","last_clean":"2024-05-26T12:02:08.538361+0000","last_became_active":"2024-05-26T12:00:28.672626+0000","last_became_peered":"2024-05-26T12:00:28.672626+0000","last_unstale":"2024-05-26T12:02:08.538361+0000","last_undegraded":"2024-05-26T12:02:08.538361+0000","last_fullsized":"2024-05-26T12:02:08.538361+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:34:31.449986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"79'1","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579574+0000","last_change":"2024-05-26T12:00:32.622325+0000","last_active":"2024-05-26T12:02:08.579574+0000","last_peered":"2024-05-26T12:02:08.579574+0000","last_clean":"2024-05-26T12:02:08.579574+0000","last_became_active":"2024-05-26T12:00:32.621970+0000","last_became_peered":"2024-05-26T12:00:32.621970+0000","last_unstale":"2024-05-26T12:02:08.579574+0000","last_undegraded":"2024-05-26T12:02:08.579574+0000","last_fullsized":"2024-05-26T12:02:08.579574+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:32:36.252057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"79'6","reported_seq":138,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:30.684045+0000","last_change":"2024-05-26T12:00:28.580235+0000","last_active":"2024-05-26T12:02:30.684045+0000","last_peered":"2024-05-26T12:02:30.684045+0000","last_clean":"2024-05-26T12:02:30.684045+0000","last_became_active":"2024-05-26T12:00:28.579812+0000","last_became_peered":"2024-05-26T12:00:28.579812+0000","last_unstale":"2024-05-26T12:02:30.684045+0000","last_undegraded":"2024-05-26T12:02:30.684045+0000","last_fullsized":"2024-05-26T12:02:30.684045+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:10:06.107316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":102,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.579380+0000","last_change":"2024-05-26T12:00:32.625187+0000","last_active":"2024-05-26T12:02:08.579380+0000","last_peered":"2024-05-26T12:02:08.579380+0000","last_clean":"2024-05-26T12:02:08.579380+0000","last_became_active":"2024-05-26T12:00:32.624833+0000","last_became_peered":"2024-05-26T12:00:32.624833+0000","last_unstale":"2024-05-26T12:02:08.579380+0000","last_undegraded":"2024-05-26T12:02:08.579380+0000","last_fullsized":"2024-05-26T12:02:08.579380+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:49:06.502017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582328+0000","last_change":"2024-05-26T12:01:07.950791+0000","last_active":"2024-05-26T12:02:08.582328+0000","last_peered":"2024-05-26T12:02:08.582328+0000","last_clean":"2024-05-26T12:02:08.582328+0000","last_became_active":"2024-05-26T12:01:07.950397+0000","last_became_peered":"2024-05-26T12:01:07.950397+0000","last_unstale":"2024-05-26T12:02:08.582328+0000","last_undegraded":"2024-05-26T12:02:08.582328+0000","last_fullsized":"2024-05-26T12:02:08.582328+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:20:15.193328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"79'5","reported_seq":141,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:35.669580+0000","last_change":"2024-05-26T12:00:28.671869+0000","last_active":"2024-05-26T12:02:35.669580+0000","last_peered":"2024-05-26T12:02:35.669580+0000","last_clean":"2024-05-26T12:02:35.669580+0000","last_became_active":"2024-05-26T12:00:28.671529+0000","last_became_peered":"2024-05-26T12:00:28.671529+0000","last_unstale":"2024-05-26T12:02:35.669580+0000","last_undegraded":"2024-05-26T12:02:35.669580+0000","last_fullsized":"2024-05-26T12:02:35.669580+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:30:17.224354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":105,"num_read_kb":105,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536965+0000","last_change":"2024-05-26T12:00:32.621164+0000","last_active":"2024-05-26T12:02:08.536965+0000","last_peered":"2024-05-26T12:02:08.536965+0000","last_clean":"2024-05-26T12:02:08.536965+0000","last_became_active":"2024-05-26T12:00:32.618422+0000","last_became_peered":"2024-05-26T12:00:32.618422+0000","last_unstale":"2024-05-26T12:02:08.536965+0000","last_undegraded":"2024-05-26T12:02:08.536965+0000","last_fullsized":"2024-05-26T12:02:08.536965+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:33:29.567862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"79'6","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.621764+0000","last_change":"2024-05-26T12:00:28.579949+0000","last_active":"2024-05-26T12:02:08.621764+0000","last_peered":"2024-05-26T12:02:08.621764+0000","last_clean":"2024-05-26T12:02:08.621764+0000","last_became_active":"2024-05-26T12:00:28.579587+0000","last_became_peered":"2024-05-26T12:00:28.579587+0000","last_unstale":"2024-05-26T12:02:08.621764+0000","last_undegraded":"2024-05-26T12:02:08.621764+0000","last_fullsized":"2024-05-26T12:02:08.621764+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:26:18.491139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'92","reported_seq":104,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639976+0000","last_change":"2024-05-26T11:59:02.705869+0000","last_active":"2024-05-26T12:02:08.639976+0000","last_peered":"2024-05-26T12:02:08.639976+0000","last_clean":"2024-05-26T12:02:08.639976+0000","last_became_active":"2024-05-26T11:59:02.700918+0000","last_became_peered":"2024-05-26T11:59:02.700918+0000","last_unstale":"2024-05-26T12:02:08.639976+0000","last_undegraded":"2024-05-26T12:02:08.639976+0000","last_fullsized":"2024-05-26T12:02:08.639976+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T11:55:35.143931+0000","last_clean_scrub_stamp":"2024-05-26T11:55:35.143931+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:01:24.735040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618717+0000","last_change":"2024-05-26T12:00:32.606843+0000","last_active":"2024-05-26T12:02:08.618717+0000","last_peered":"2024-05-26T12:02:08.618717+0000","last_clean":"2024-05-26T12:02:08.618717+0000","last_became_active":"2024-05-26T12:00:32.606552+0000","last_became_peered":"2024-05-26T12:00:32.606552+0000","last_unstale":"2024-05-26T12:02:08.618717+0000","last_undegraded":"2024-05-26T12:02:08.618717+0000","last_fullsized":"2024-05-26T12:02:08.618717+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:43:03.113199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640311+0000","last_change":"2024-05-26T12:00:28.581819+0000","last_active":"2024-05-26T12:02:08.640311+0000","last_peered":"2024-05-26T12:02:08.640311+0000","last_clean":"2024-05-26T12:02:08.640311+0000","last_became_active":"2024-05-26T12:00:28.581430+0000","last_became_peered":"2024-05-26T12:00:28.581430+0000","last_unstale":"2024-05-26T12:02:08.640311+0000","last_undegraded":"2024-05-26T12:02:08.640311+0000","last_fullsized":"2024-05-26T12:02:08.640311+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:13:42.556238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"79'4","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640592+0000","last_change":"2024-05-26T12:00:28.672396+0000","last_active":"2024-05-26T12:02:08.640592+0000","last_peered":"2024-05-26T12:02:08.640592+0000","last_clean":"2024-05-26T12:02:08.640592+0000","last_became_active":"2024-05-26T12:00:28.672045+0000","last_became_peered":"2024-05-26T12:00:28.672045+0000","last_unstale":"2024-05-26T12:02:08.640592+0000","last_undegraded":"2024-05-26T12:02:08.640592+0000","last_fullsized":"2024-05-26T12:02:08.640592+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:02:25.925210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.618752+0000","last_change":"2024-05-26T12:00:32.606876+0000","last_active":"2024-05-26T12:02:08.618752+0000","last_peered":"2024-05-26T12:02:08.618752+0000","last_clean":"2024-05-26T12:02:08.618752+0000","last_became_active":"2024-05-26T12:00:32.606161+0000","last_became_peered":"2024-05-26T12:00:32.606161+0000","last_unstale":"2024-05-26T12:02:08.618752+0000","last_undegraded":"2024-05-26T12:02:08.618752+0000","last_fullsized":"2024-05-26T12:02:08.618752+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:47:02.451447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"81'15","reported_seq":61,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538445+0000","last_change":"2024-05-26T12:01:07.955816+0000","last_active":"2024-05-26T12:02:08.538445+0000","last_peered":"2024-05-26T12:02:08.538445+0000","last_clean":"2024-05-26T12:02:08.538445+0000","last_became_active":"2024-05-26T12:01:07.947443+0000","last_became_peered":"2024-05-26T12:01:07.947443+0000","last_unstale":"2024-05-26T12:02:08.538445+0000","last_undegraded":"2024-05-26T12:02:08.538445+0000","last_fullsized":"2024-05-26T12:02:08.538445+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:30:00.005339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.642399+0000","last_change":"2024-05-26T12:02:08.642045+0000","last_active":"2024-05-26T12:02:08.642399+0000","last_peered":"2024-05-26T12:02:08.642399+0000","last_clean":"2024-05-26T12:02:08.642399+0000","last_became_active":"2024-05-26T12:02:08.641734+0000","last_became_peered":"2024-05-26T12:02:08.641734+0000","last_unstale":"2024-05-26T12:02:08.642399+0000","last_undegraded":"2024-05-26T12:02:08.642399+0000","last_fullsized":"2024-05-26T12:02:08.642399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:19:09.909218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,7],"acting":[9,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.7","version":"79'7","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.622031+0000","last_change":"2024-05-26T12:00:28.971347+0000","last_active":"2024-05-26T12:02:08.622031+0000","last_peered":"2024-05-26T12:02:08.622031+0000","last_clean":"2024-05-26T12:02:08.622031+0000","last_became_active":"2024-05-26T12:00:28.970911+0000","last_became_peered":"2024-05-26T12:00:28.970911+0000","last_unstale":"2024-05-26T12:02:08.622031+0000","last_undegraded":"2024-05-26T12:02:08.622031+0000","last_fullsized":"2024-05-26T12:02:08.622031+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:47:45.306875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.834913+0000","last_change":"2024-05-26T12:00:32.614238+0000","last_active":"2024-05-26T12:02:08.834913+0000","last_peered":"2024-05-26T12:02:08.834913+0000","last_clean":"2024-05-26T12:02:08.834913+0000","last_became_active":"2024-05-26T12:00:32.607018+0000","last_became_peered":"2024-05-26T12:00:32.607018+0000","last_unstale":"2024-05-26T12:02:08.834913+0000","last_undegraded":"2024-05-26T12:02:08.834913+0000","last_fullsized":"2024-05-26T12:02:08.834913+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:31:06.250615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"79'2","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538122+0000","last_change":"2024-05-26T12:00:28.671961+0000","last_active":"2024-05-26T12:02:08.538122+0000","last_peered":"2024-05-26T12:02:08.538122+0000","last_clean":"2024-05-26T12:02:08.538122+0000","last_became_active":"2024-05-26T12:00:28.671672+0000","last_became_peered":"2024-05-26T12:00:28.671672+0000","last_unstale":"2024-05-26T12:02:08.538122+0000","last_undegraded":"2024-05-26T12:02:08.538122+0000","last_fullsized":"2024-05-26T12:02:08.538122+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:31:05.646044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639950+0000","last_change":"2024-05-26T12:00:32.611432+0000","last_active":"2024-05-26T12:02:08.639950+0000","last_peered":"2024-05-26T12:02:08.639950+0000","last_clean":"2024-05-26T12:02:08.639950+0000","last_became_active":"2024-05-26T12:00:32.610686+0000","last_became_peered":"2024-05-26T12:00:32.610686+0000","last_unstale":"2024-05-26T12:02:08.639950+0000","last_undegraded":"2024-05-26T12:02:08.639950+0000","last_fullsized":"2024-05-26T12:02:08.639950+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:52:36.459552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"79'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536814+0000","last_change":"2024-05-26T12:00:28.576723+0000","last_active":"2024-05-26T12:02:08.536814+0000","last_peered":"2024-05-26T12:02:08.536814+0000","last_clean":"2024-05-26T12:02:08.536814+0000","last_became_active":"2024-05-26T12:00:28.576380+0000","last_became_peered":"2024-05-26T12:00:28.576380+0000","last_unstale":"2024-05-26T12:02:08.536814+0000","last_undegraded":"2024-05-26T12:02:08.536814+0000","last_fullsized":"2024-05-26T12:02:08.536814+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:49:25.611830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640945+0000","last_change":"2024-05-26T12:00:32.621124+0000","last_active":"2024-05-26T12:02:08.640945+0000","last_peered":"2024-05-26T12:02:08.640945+0000","last_clean":"2024-05-26T12:02:08.640945+0000","last_became_active":"2024-05-26T12:00:32.620740+0000","last_became_peered":"2024-05-26T12:00:32.620740+0000","last_unstale":"2024-05-26T12:02:08.640945+0000","last_undegraded":"2024-05-26T12:02:08.640945+0000","last_fullsized":"2024-05-26T12:02:08.640945+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:15:16.123821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"79'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537934+0000","last_change":"2024-05-26T12:00:28.572222+0000","last_active":"2024-05-26T12:02:08.537934+0000","last_peered":"2024-05-26T12:02:08.537934+0000","last_clean":"2024-05-26T12:02:08.537934+0000","last_became_active":"2024-05-26T12:00:28.571459+0000","last_became_peered":"2024-05-26T12:00:28.571459+0000","last_unstale":"2024-05-26T12:02:08.537934+0000","last_undegraded":"2024-05-26T12:02:08.537934+0000","last_fullsized":"2024-05-26T12:02:08.537934+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:49:56.234550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537165+0000","last_change":"2024-05-26T12:00:32.618066+0000","last_active":"2024-05-26T12:02:08.537165+0000","last_peered":"2024-05-26T12:02:08.537165+0000","last_clean":"2024-05-26T12:02:08.537165+0000","last_became_active":"2024-05-26T12:00:32.617579+0000","last_became_peered":"2024-05-26T12:00:32.617579+0000","last_unstale":"2024-05-26T12:02:08.537165+0000","last_undegraded":"2024-05-26T12:02:08.537165+0000","last_fullsized":"2024-05-26T12:02:08.537165+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:46:57.045155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"79'4","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640965+0000","last_change":"2024-05-26T12:00:28.584555+0000","last_active":"2024-05-26T12:02:08.640965+0000","last_peered":"2024-05-26T12:02:08.640965+0000","last_clean":"2024-05-26T12:02:08.640965+0000","last_became_active":"2024-05-26T12:00:28.583750+0000","last_became_peered":"2024-05-26T12:00:28.583750+0000","last_unstale":"2024-05-26T12:02:08.640965+0000","last_undegraded":"2024-05-26T12:02:08.640965+0000","last_fullsized":"2024-05-26T12:02:08.640965+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T22:26:50.312369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639766+0000","last_change":"2024-05-26T12:00:32.616082+0000","last_active":"2024-05-26T12:02:08.639766+0000","last_peered":"2024-05-26T12:02:08.639766+0000","last_clean":"2024-05-26T12:02:08.639766+0000","last_became_active":"2024-05-26T12:00:32.615758+0000","last_became_peered":"2024-05-26T12:00:32.615758+0000","last_unstale":"2024-05-26T12:02:08.639766+0000","last_undegraded":"2024-05-26T12:02:08.639766+0000","last_fullsized":"2024-05-26T12:02:08.639766+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:17:52.776655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"79'5","reported_seq":139,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:32.671977+0000","last_change":"2024-05-26T12:00:28.971186+0000","last_active":"2024-05-26T12:02:32.671977+0000","last_peered":"2024-05-26T12:02:32.671977+0000","last_clean":"2024-05-26T12:02:32.671977+0000","last_became_active":"2024-05-26T12:00:28.970706+0000","last_became_peered":"2024-05-26T12:00:28.970706+0000","last_unstale":"2024-05-26T12:02:32.671977+0000","last_undegraded":"2024-05-26T12:02:32.671977+0000","last_fullsized":"2024-05-26T12:02:32.671977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:39:12.139262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":104,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.639815+0000","last_change":"2024-05-26T12:00:32.623244+0000","last_active":"2024-05-26T12:02:08.639815+0000","last_peered":"2024-05-26T12:02:08.639815+0000","last_clean":"2024-05-26T12:02:08.639815+0000","last_became_active":"2024-05-26T12:00:32.622057+0000","last_became_peered":"2024-05-26T12:00:32.622057+0000","last_unstale":"2024-05-26T12:02:08.639815+0000","last_undegraded":"2024-05-26T12:02:08.639815+0000","last_fullsized":"2024-05-26T12:02:08.639815+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T15:37:09.871626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"79'3","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.582143+0000","last_change":"2024-05-26T12:00:28.576115+0000","last_active":"2024-05-26T12:02:08.582143+0000","last_peered":"2024-05-26T12:02:08.582143+0000","last_clean":"2024-05-26T12:02:08.582143+0000","last_became_active":"2024-05-26T12:00:28.575704+0000","last_became_peered":"2024-05-26T12:00:28.575704+0000","last_unstale":"2024-05-26T12:02:08.582143+0000","last_undegraded":"2024-05-26T12:02:08.582143+0000","last_fullsized":"2024-05-26T12:02:08.582143+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:28:05.142390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"81'1","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640027+0000","last_change":"2024-05-26T12:00:32.611997+0000","last_active":"2024-05-26T12:02:08.640027+0000","last_peered":"2024-05-26T12:02:08.640027+0000","last_clean":"2024-05-26T12:02:08.640027+0000","last_became_active":"2024-05-26T12:00:32.611646+0000","last_became_peered":"2024-05-26T12:00:32.611646+0000","last_unstale":"2024-05-26T12:02:08.640027+0000","last_undegraded":"2024-05-26T12:02:08.640027+0000","last_fullsized":"2024-05-26T12:02:08.640027+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:50:15.856048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"79'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619497+0000","last_change":"2024-05-26T12:00:28.576500+0000","last_active":"2024-05-26T12:02:08.619497+0000","last_peered":"2024-05-26T12:02:08.619497+0000","last_clean":"2024-05-26T12:02:08.619497+0000","last_became_active":"2024-05-26T12:00:28.576171+0000","last_became_peered":"2024-05-26T12:00:28.576171+0000","last_unstale":"2024-05-26T12:02:08.619497+0000","last_undegraded":"2024-05-26T12:02:08.619497+0000","last_fullsized":"2024-05-26T12:02:08.619497+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:27.556295+0000","last_clean_scrub_stamp":"2024-05-26T12:00:27.556295+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:44:54.786997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640451+0000","last_change":"2024-05-26T12:00:32.617550+0000","last_active":"2024-05-26T12:02:08.640451+0000","last_peered":"2024-05-26T12:02:08.640451+0000","last_clean":"2024-05-26T12:02:08.640451+0000","last_became_active":"2024-05-26T12:00:32.616821+0000","last_became_peered":"2024-05-26T12:00:32.616821+0000","last_unstale":"2024-05-26T12:02:08.640451+0000","last_undegraded":"2024-05-26T12:02:08.640451+0000","last_fullsized":"2024-05-26T12:02:08.640451+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:49:59.956694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640212+0000","last_change":"2024-05-26T12:00:32.623314+0000","last_active":"2024-05-26T12:02:08.640212+0000","last_peered":"2024-05-26T12:02:08.640212+0000","last_clean":"2024-05-26T12:02:08.640212+0000","last_became_active":"2024-05-26T12:00:32.622187+0000","last_became_peered":"2024-05-26T12:00:32.622187+0000","last_unstale":"2024-05-26T12:02:08.640212+0000","last_undegraded":"2024-05-26T12:02:08.640212+0000","last_fullsized":"2024-05-26T12:02:08.640212+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T13:19:08.026803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537778+0000","last_change":"2024-05-26T12:00:32.614814+0000","last_active":"2024-05-26T12:02:08.537778+0000","last_peered":"2024-05-26T12:02:08.537778+0000","last_clean":"2024-05-26T12:02:08.537778+0000","last_became_active":"2024-05-26T12:00:32.614439+0000","last_became_peered":"2024-05-26T12:00:32.614439+0000","last_unstale":"2024-05-26T12:02:08.537778+0000","last_undegraded":"2024-05-26T12:02:08.537778+0000","last_fullsized":"2024-05-26T12:02:08.537778+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:56:10.817914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537746+0000","last_change":"2024-05-26T12:00:32.610170+0000","last_active":"2024-05-26T12:02:08.537746+0000","last_peered":"2024-05-26T12:02:08.537746+0000","last_clean":"2024-05-26T12:02:08.537746+0000","last_became_active":"2024-05-26T12:00:32.609783+0000","last_became_peered":"2024-05-26T12:00:32.609783+0000","last_unstale":"2024-05-26T12:02:08.537746+0000","last_undegraded":"2024-05-26T12:02:08.537746+0000","last_fullsized":"2024-05-26T12:02:08.537746+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T19:33:52.788561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.835474+0000","last_change":"2024-05-26T12:00:32.609819+0000","last_active":"2024-05-26T12:02:08.835474+0000","last_peered":"2024-05-26T12:02:08.835474+0000","last_clean":"2024-05-26T12:02:08.835474+0000","last_became_active":"2024-05-26T12:00:32.609016+0000","last_became_peered":"2024-05-26T12:00:32.609016+0000","last_unstale":"2024-05-26T12:02:08.835474+0000","last_undegraded":"2024-05-26T12:02:08.835474+0000","last_fullsized":"2024-05-26T12:02:08.835474+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:10:09.582603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619395+0000","last_change":"2024-05-26T12:00:32.608904+0000","last_active":"2024-05-26T12:02:08.619395+0000","last_peered":"2024-05-26T12:02:08.619395+0000","last_clean":"2024-05-26T12:02:08.619395+0000","last_became_active":"2024-05-26T12:00:32.608505+0000","last_became_peered":"2024-05-26T12:00:32.608505+0000","last_unstale":"2024-05-26T12:02:08.619395+0000","last_undegraded":"2024-05-26T12:02:08.619395+0000","last_fullsized":"2024-05-26T12:02:08.619395+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T18:34:09.931046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640467+0000","last_change":"2024-05-26T12:00:32.618370+0000","last_active":"2024-05-26T12:02:08.640467+0000","last_peered":"2024-05-26T12:02:08.640467+0000","last_clean":"2024-05-26T12:02:08.640467+0000","last_became_active":"2024-05-26T12:00:32.618022+0000","last_became_peered":"2024-05-26T12:00:32.618022+0000","last_unstale":"2024-05-26T12:02:08.640467+0000","last_undegraded":"2024-05-26T12:02:08.640467+0000","last_fullsized":"2024-05-26T12:02:08.640467+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T16:45:12.652460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.537935+0000","last_change":"2024-05-26T12:00:32.612319+0000","last_active":"2024-05-26T12:02:08.537935+0000","last_peered":"2024-05-26T12:02:08.537935+0000","last_clean":"2024-05-26T12:02:08.537935+0000","last_became_active":"2024-05-26T12:00:32.611973+0000","last_became_peered":"2024-05-26T12:00:32.611973+0000","last_unstale":"2024-05-26T12:02:08.537935+0000","last_undegraded":"2024-05-26T12:02:08.537935+0000","last_fullsized":"2024-05-26T12:02:08.537935+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T21:23:48.261359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536954+0000","last_change":"2024-05-26T12:00:32.613461+0000","last_active":"2024-05-26T12:02:08.536954+0000","last_peered":"2024-05-26T12:02:08.536954+0000","last_clean":"2024-05-26T12:02:08.536954+0000","last_became_active":"2024-05-26T12:00:32.613068+0000","last_became_peered":"2024-05-26T12:00:32.613068+0000","last_unstale":"2024-05-26T12:02:08.536954+0000","last_undegraded":"2024-05-26T12:02:08.536954+0000","last_fullsized":"2024-05-26T12:02:08.536954+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:17:59.606664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.581931+0000","last_change":"2024-05-26T12:00:32.615583+0000","last_active":"2024-05-26T12:02:08.581931+0000","last_peered":"2024-05-26T12:02:08.581931+0000","last_clean":"2024-05-26T12:02:08.581931+0000","last_became_active":"2024-05-26T12:00:32.611502+0000","last_became_peered":"2024-05-26T12:00:32.611502+0000","last_unstale":"2024-05-26T12:02:08.581931+0000","last_undegraded":"2024-05-26T12:02:08.581931+0000","last_fullsized":"2024-05-26T12:02:08.581931+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T23:26:47.772010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.536763+0000","last_change":"2024-05-26T12:00:32.612190+0000","last_active":"2024-05-26T12:02:08.536763+0000","last_peered":"2024-05-26T12:02:08.536763+0000","last_clean":"2024-05-26T12:02:08.536763+0000","last_became_active":"2024-05-26T12:00:32.611843+0000","last_became_peered":"2024-05-26T12:00:32.611843+0000","last_unstale":"2024-05-26T12:02:08.536763+0000","last_undegraded":"2024-05-26T12:02:08.536763+0000","last_fullsized":"2024-05-26T12:02:08.536763+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T17:23:05.244741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.538393+0000","last_change":"2024-05-26T12:00:32.620758+0000","last_active":"2024-05-26T12:02:08.538393+0000","last_peered":"2024-05-26T12:02:08.538393+0000","last_clean":"2024-05-26T12:02:08.538393+0000","last_became_active":"2024-05-26T12:00:32.620401+0000","last_became_peered":"2024-05-26T12:00:32.620401+0000","last_unstale":"2024-05-26T12:02:08.538393+0000","last_undegraded":"2024-05-26T12:02:08.538393+0000","last_fullsized":"2024-05-26T12:02:08.538393+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T20:59:18.097154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.619247+0000","last_change":"2024-05-26T12:00:32.615615+0000","last_active":"2024-05-26T12:02:08.619247+0000","last_peered":"2024-05-26T12:02:08.619247+0000","last_clean":"2024-05-26T12:02:08.619247+0000","last_became_active":"2024-05-26T12:00:32.615304+0000","last_became_peered":"2024-05-26T12:00:32.615304+0000","last_unstale":"2024-05-26T12:02:08.619247+0000","last_undegraded":"2024-05-26T12:02:08.619247+0000","last_fullsized":"2024-05-26T12:02:08.619247+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T12:05:22.568910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-05-26T12:02:08.640390+0000","last_change":"2024-05-26T12:00:32.627209+0000","last_active":"2024-05-26T12:02:08.640390+0000","last_peered":"2024-05-26T12:02:08.640390+0000","last_clean":"2024-05-26T12:02:08.640390+0000","last_became_active":"2024-05-26T12:00:32.626851+0000","last_became_peered":"2024-05-26T12:00:32.626851+0000","last_unstale":"2024-05-26T12:02:08.640390+0000","last_undegraded":"2024-05-26T12:02:08.640390+0000","last_fullsized":"2024-05-26T12:02:08.640390+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":76,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-26T12:00:31.580486+0000","last_clean_scrub_stamp":"2024-05-26T12:00:31.580486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-27T14:05:40.754565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213552,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66575,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1054,"num_read_kb":596,"num_write":244,"num_write_kb":134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":28173,"num_keys_recovered":17,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":548864,"data_stored":199725,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678057,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":888,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":909312,"data_stored":445991,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874286,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":1096692,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":445831,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234041,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":446209,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430270,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":978944,"data_stored":1038727,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":43,"seq":184683593796,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27912,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717240,"statfs":{"total":95995035648,"available":95966453760,"internally_reserved":0,"allocated":913408,"data_stored":445939,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574941,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":506614,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738465,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36256,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708896,"statfs":{"total":95995035648,"available":95957909504,"internally_reserved":0,"allocated":933888,"data_stored":446887,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411416,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":446681,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149779,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":445373,"data_compressed":11570,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953550,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32128,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713024,"statfs":{"total":95995035648,"available":95962136576,"internally_reserved":0,"allocated":966656,"data_stored":505171,"data_compressed":16130,"data_compressed_allocated":24576,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":37,"seq":158913790024,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":1037924,"data_compressed":47702,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34512,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36132,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61905,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":60462,"data_compressed":4560,"data_compressed_allocated":8192,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-26T12:02:37.971 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-26T12:02:37.971 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-26T12:02:37.971 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-26T12:02:37.971 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph health --format=json 2024-05-26T12:02:38.259 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:38.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:02:38.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:38 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:02:38.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:38 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: pgmap v358: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:02:38.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:38 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 -- 172.21.15.156:0/4205312830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1006bc90 msgr2=0x7f8f1006c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 --2- 172.21.15.156:0/4205312830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1006bc90 0x7f8f1006c110 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f8ef8009990 tx=0x7f8ef8009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 -- 172.21.15.156:0/4205312830 shutdown_connections 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 --2- 172.21.15.156:0/4205312830 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006c650 0x7f8f101103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 --2- 172.21.15.156:0/4205312830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f101095b0 0x7f8f10109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 --2- 172.21.15.156:0/4205312830 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f1006bc90 0x7f8f1006c110 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.014 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 -- 172.21.15.156:0/4205312830 >> 172.21.15.156:0/4205312830 conn(0x7f8f10079410 msgr2=0x7f8f10079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:40.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 -- 172.21.15.156:0/4205312830 shutdown_connections 2024-05-26T12:02:40.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 -- 172.21.15.156:0/4205312830 wait complete. 2024-05-26T12:02:40.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.013+0000 7f8f15c09700 1 Processor -- start 2024-05-26T12:02:40.015 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 -- start start 2024-05-26T12:02:40.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 0x7f8f1019f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:40.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 0x7f8f1019fdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:40.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:40.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8f10113740 con 0x7f8f101095b0 2024-05-26T12:02:40.016 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8f101135c0 con 0x7f8f1006c650 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f15c09700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8f101138c0 con 0x7f8f1006bc90 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51784/0 (socket says 172.21.15.156:51784) 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 -- 172.21.15.156:0/2324688041 learned_addr learned my addr 172.21.15.156:0/2324688041 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0f7fe700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 0x7f8f1019f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0effd700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 0x7f8f1019fdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 -- 172.21.15.156:0/2324688041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 msgr2=0x7f8f1019f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 0x7f8f1019f890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 -- 172.21.15.156:0/2324688041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 msgr2=0x7f8f1019fdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 0x7f8f1019fdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0ffff700 1 -- 172.21.15.156:0/2324688041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f1019a200 con 0x7f8f101095b0 2024-05-26T12:02:40.017 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0effd700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 0x7f8f1019fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.014+0000 7f8f0f7fe700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 0x7f8f1019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f0ffff700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b550 tx=0x7f8f0800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08015020 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f1019a4f0 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f08011ce0 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f08010d40 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.015+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f101a6760 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.016+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f1006a230 con 0x7f8f101095b0 2024-05-26T12:02:40.018 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.018+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f8f08018020 con 0x7f8f101095b0 2024-05-26T12:02:40.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.019+0000 7f8f0cff9700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 0x7f8efc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:40.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.019+0000 7f8f0f7fe700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 0x7f8efc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:40.019 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.019+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f8f080e1f80 con 0x7f8f101095b0 2024-05-26T12:02:40.020 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.019+0000 7f8f0f7fe700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 0x7f8efc0b4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f10100aa0 tx=0x7f8f00009040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:40.022 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.022+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8f0801f020 con 0x7f8f101095b0 2024-05-26T12:02:40.271 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.271+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f8f1019afb0 con 0x7f8f101095b0 2024-05-26T12:02:40.272 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:40 smithi156 ceph-mon[32025]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-26T12:02:40.272 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.271+0000 7f8f0cff9700 1 -- 172.21.15.156:0/2324688041 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f8f1019afb0 con 0x7f8f101095b0 2024-05-26T12:02:40.272 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:40.272 INFO:teuthology.orchestra.run.smithi156.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 msgr2=0x7f8efc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 0x7f8efc0b4d40 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8f10100aa0 tx=0x7f8f00009040 comp rx=0 tx=0).stop 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 msgr2=0x7f8f10199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f8f0800b550 tx=0x7f8f0800b910 comp rx=0 tx=0).stop 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 shutdown_connections 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8efc0b2880 0x7f8efc0b4d40 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.274 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8f101095b0 0x7f8f10199aa0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8f1006bc90 0x7f8f1019f890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 --2- 172.21.15.156:0/2324688041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8f1006c650 0x7f8f1019fdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:40.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.273+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 >> 172.21.15.156:0/2324688041 conn(0x7f8f10079410 msgr2=0x7f8f10101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:40.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.274+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 shutdown_connections 2024-05-26T12:02:40.275 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:40.274+0000 7f8f15c09700 1 -- 172.21.15.156:0/2324688041 wait complete. 2024-05-26T12:02:40.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:40 smithi192 ceph-mon[36545]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-26T12:02:40.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:40 smithi186 ceph-mon[36412]: pgmap v359: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-05-26T12:02:40.971 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-26T12:02:40.972 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:02:41.214 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:41.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:41 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2324688041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-26T12:02:41.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:41 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2324688041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-26T12:02:41.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:41 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2324688041' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-26T12:02:42.295 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:42 smithi156 ceph-mon[32025]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:02:42.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:42 smithi192 ceph-mon[36545]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:02:42.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:42 smithi186 ceph-mon[36412]: pgmap v360: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 426 B/s wr, 12 op/s 2024-05-26T12:02:42.883 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- 172.21.15.156:0/280201828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030075770 msgr2=0x7fa030075b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 --2- 172.21.15.156:0/280201828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030075770 0x7fa030075b50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa024009990 tx=0x7fa024009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- 172.21.15.156:0/280201828 shutdown_connections 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 --2- 172.21.15.156:0/280201828 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 0x7fa030116b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 --2- 172.21.15.156:0/280201828 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030075770 0x7fa030075b50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 --2- 172.21.15.156:0/280201828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030076090 0x7fa0301141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- 172.21.15.156:0/280201828 >> 172.21.15.156:0/280201828 conn(0x7fa03007a6a0 msgr2=0x7fa03007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- 172.21.15.156:0/280201828 shutdown_connections 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- 172.21.15.156:0/280201828 wait complete. 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 Processor -- start 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.881+0000 7fa036dd6700 1 -- start start 2024-05-26T12:02:42.884 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030075770 0x7fa0301a0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 0x7fa0301a4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fa030119d70 con 0x7fa030076090 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fa030119bf0 con 0x7fa030075770 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fa030119ef0 con 0x7fa030114710 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51808/0 (socket says 172.21.15.156:51808) 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 -- 172.21.15.156:0/30852126 learned_addr learned my addr 172.21.15.156:0/30852126 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa034b72700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030075770 0x7fa0301a0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa035373700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 0x7fa0301a4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:42.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 -- 172.21.15.156:0/30852126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 msgr2=0x7fa0301a4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 0x7fa0301a4b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 -- 172.21.15.156:0/30852126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030075770 msgr2=0x7fa0301a0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030075770 0x7fa0301a0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 -- 172.21.15.156:0/30852126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0301a5250 con 0x7fa030076090 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02ffff700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00d950 tx=0x7fa01c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c016070 con 0x7fa030076090 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa01c009df0 con 0x7fa030076090 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa01c00f3c0 con 0x7fa030076090 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.882+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0301a5540 con 0x7fa030076090 2024-05-26T12:02:42.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.883+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0301acdc0 con 0x7fa030076090 2024-05-26T12:02:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.884+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa03006a230 con 0x7fa030076090 2024-05-26T12:02:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.885+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fa01c019070 con 0x7fa030076090 2024-05-26T12:02:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.886+0000 7fa02dffb700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 0x7fa0200b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.886+0000 7fa034b72700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 0x7fa0200b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:42.887 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.886+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fa01c0e1e60 con 0x7fa030076090 2024-05-26T12:02:42.888 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.886+0000 7fa034b72700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 0x7fa0200b4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa02400b5c0 tx=0x7fa024005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:42.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:42.891+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fa01c0a9b50 con 0x7fa030076090 2024-05-26T12:02:43.142 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.142+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa0301acfc0 con 0x7fa030076090 2024-05-26T12:02:43.143 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.142+0000 7fa02dffb700 1 -- 172.21.15.156:0/30852126 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7fa01c0a9450 con 0x7fa030076090 2024-05-26T12:02:43.143 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:43.143 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 msgr2=0x7fa0200b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 0x7fa0200b4d40 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa02400b5c0 tx=0x7fa024005960 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 msgr2=0x7fa0301a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa01c00d950 tx=0x7fa01c00dc60 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 shutdown_connections 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fa0200b2880 0x7fa0200b4d40 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fa030114710 0x7fa0301a4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa030075770 0x7fa0301a0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 --2- 172.21.15.156:0/30852126 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fa030076090 0x7fa0301a0780 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 >> 172.21.15.156:0/30852126 conn(0x7fa03007a6a0 msgr2=0x7fa0301101b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:43.146 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 shutdown_connections 2024-05-26T12:02:43.147 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:43.145+0000 7fa036dd6700 1 -- 172.21.15.156:0/30852126 wait complete. 2024-05-26T12:02:43.147 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:02:43.518 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:43 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/30852126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:43.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:43 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/30852126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:43.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:43 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/30852126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:43.873 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:02:44.134 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:44.397 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:44 smithi156 ceph-mon[32025]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:44.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:44 smithi192 ceph-mon[36545]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:44.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:44 smithi186 ceph-mon[36412]: pgmap v361: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 -- 172.21.15.156:0/895085430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec108930 msgr2=0x7febec1104b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 --2- 172.21.15.156:0/895085430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec108930 0x7febec1104b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7febe400b0d0 tx=0x7febe400b3e0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 -- 172.21.15.156:0/895085430 shutdown_connections 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 --2- 172.21.15.156:0/895085430 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec108930 0x7febec1104b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 --2- 172.21.15.156:0/895085430 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec101cf0 0x7febec1020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 --2- 172.21.15.156:0/895085430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 0x7febec108200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 -- 172.21.15.156:0/895085430 >> 172.21.15.156:0/895085430 conn(0x7febec078a40 msgr2=0x7febec0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:45.732 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 -- 172.21.15.156:0/895085430 shutdown_connections 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.725+0000 7febf20e7700 1 -- 172.21.15.156:0/895085430 wait complete. 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 Processor -- start 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 -- start start 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 0x7febec19fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 0x7febec199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7febec113760 con 0x7febec101cf0 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7febec1135e0 con 0x7febec1026a0 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febf20e7700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7febec1138e0 con 0x7febec108930 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febeb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febeb7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51824/0 (socket says 172.21.15.156:51824) 2024-05-26T12:02:45.733 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.726+0000 7febeb7fe700 1 -- 172.21.15.156:0/2038398461 learned_addr learned my addr 172.21.15.156:0/2038398461 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeaffd700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 0x7febec19fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febebfff700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 0x7febec199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 -- 172.21.15.156:0/2038398461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 msgr2=0x7febec199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 0x7febec199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 -- 172.21.15.156:0/2038398461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 msgr2=0x7febec19fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 0x7febec19fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 -- 172.21.15.156:0/2038398461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febec199fe0 con 0x7febec101cf0 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febebfff700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 0x7febec199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febeb7fe700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c9b0 tx=0x7febdc00ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc007b00 con 0x7febec101cf0 2024-05-26T12:02:45.734 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febec19a2d0 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7febdc007ca0 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febdc018660 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.727+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febec1a6590 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.728+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febec06a230 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.730+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7febdc00f450 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.730+0000 7febe8ff9700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 0x7febd40b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.730+0000 7febeaffd700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 0x7febd40b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.730+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7febdc0e24d0 con 0x7febec101cf0 2024-05-26T12:02:45.735 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.731+0000 7febeaffd700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 0x7febd40b4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7febec101440 tx=0x7febe000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:45.736 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.734+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7febdc014030 con 0x7febec101cf0 2024-05-26T12:02:45.983 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.982+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7febec19adb0 con 0x7febec101cf0 2024-05-26T12:02:45.984 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.983+0000 7febe8ff9700 1 -- 172.21.15.156:0/2038398461 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7febdc0aa250 con 0x7febec101cf0 2024-05-26T12:02:45.984 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:45.984 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 msgr2=0x7febd40b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 0x7febd40b4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7febec101440 tx=0x7febe000a380 comp rx=0 tx=0).stop 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 msgr2=0x7febec19f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7febdc00c9b0 tx=0x7febdc00ccc0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 shutdown_connections 2024-05-26T12:02:45.986 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7febd40b2880 0x7febd40b4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7febec108930 0x7febec199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7febec101cf0 0x7febec19f6c0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 --2- 172.21.15.156:0/2038398461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febec1026a0 0x7febec19fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 >> 172.21.15.156:0/2038398461 conn(0x7febec078a40 msgr2=0x7febec106fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 shutdown_connections 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:45.985+0000 7febf20e7700 1 -- 172.21.15.156:0/2038398461 wait complete. 2024-05-26T12:02:45.987 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:02:46.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:46 smithi156 ceph-mon[32025]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-26T12:02:46.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:46 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2038398461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:46.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:46 smithi192 ceph-mon[36545]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-26T12:02:46.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:46 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2038398461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:46.702 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:02:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:46 smithi186 ceph-mon[36412]: pgmap v362: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-05-26T12:02:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:46 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2038398461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:46.949 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:48.463 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:48 smithi156 ceph-mon[32025]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-26T12:02:48.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:48 smithi192 ceph-mon[36545]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-26T12:02:48.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- 172.21.15.156:0/1608462293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 msgr2=0x7fd680103bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:48.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 --2- 172.21.15.156:0/1608462293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd680103bd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fd6700099c0 tx=0x7fd670009cd0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.686 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- 172.21.15.156:0/1608462293 shutdown_connections 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 --2- 172.21.15.156:0/1608462293 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 0x7fd680110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 --2- 172.21.15.156:0/1608462293 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd680103bd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 --2- 172.21.15.156:0/1608462293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd6801081c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- 172.21.15.156:0/1608462293 >> 172.21.15.156:0/1608462293 conn(0x7fd68007a6a0 msgr2=0x7fd68007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- 172.21.15.156:0/1608462293 shutdown_connections 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- 172.21.15.156:0/1608462293 wait complete. 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 Processor -- start 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.683+0000 7fd68766a700 1 -- start start 2024-05-26T12:02:48.687 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd68019fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 0x7fd680199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd680113740 con 0x7fd6801037f0 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801135c0 con 0x7fd6801041a0 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd6801138c0 con 0x7fd6801088f0 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51840/0 (socket says 172.21.15.156:51840) 2024-05-26T12:02:48.688 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 -- 172.21.15.156:0/3403275654 learned_addr learned my addr 172.21.15.156:0/3403275654 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd684c05700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd68019fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685c07700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 0x7fd680199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 -- 172.21.15.156:0/3403275654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 msgr2=0x7fd680199a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 0x7fd680199a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 -- 172.21.15.156:0/3403275654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 msgr2=0x7fd68019fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd68019fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 -- 172.21.15.156:0/3403275654 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd68019a160 con 0x7fd6801037f0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd684c05700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd68019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd685406700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd6700045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670026070 con 0x7fd6801037f0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd67000f460 con 0x7fd6801037f0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.684+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd68019a3f0 con 0x7fd6801037f0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.685+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd670009e60 con 0x7fd6801037f0 2024-05-26T12:02:48.689 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.685+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6801a6710 con 0x7fd6801037f0 2024-05-26T12:02:48.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.686+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68006a230 con 0x7fd6801037f0 2024-05-26T12:02:48.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.687+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fd670029070 con 0x7fd6801037f0 2024-05-26T12:02:48.690 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.687+0000 7fd6767fc700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 0x7fd66c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:48.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.688+0000 7fd684c05700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 0x7fd66c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:48.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.688+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fd6700e2a40 con 0x7fd6801037f0 2024-05-26T12:02:48.691 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.688+0000 7fd684c05700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 0x7fd66c0b4dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd680103160 tx=0x7fd67c009260 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:48.692 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.692+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fd670013070 con 0x7fd6801037f0 2024-05-26T12:02:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:48 smithi186 ceph-mon[36412]: pgmap v363: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-05-26T12:02:48.906 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.906+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd68002e560 con 0x7fd6801037f0 2024-05-26T12:02:48.908 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.907+0000 7fd6767fc700 1 -- 172.21.15.156:0/3403275654 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+22753 (secure 0 0 0) 0x7fd6700aa7c0 con 0x7fd6801037f0 2024-05-26T12:02:48.908 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:48.908 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":83,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:02:08.534113+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:02:48.910 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.909+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 msgr2=0x7fd66c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.909+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 0x7fd66c0b4dc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fd680103160 tx=0x7fd67c009260 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.909+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 msgr2=0x7fd68019f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.909+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd670006010 tx=0x7fd6700045e0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 shutdown_connections 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd66c0b2900 0x7fd66c0b4dc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd6801088f0 0x7fd680199a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd6801037f0 0x7fd68019f840 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 --2- 172.21.15.156:0/3403275654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6801041a0 0x7fd68019fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 >> 172.21.15.156:0/3403275654 conn(0x7fd68007a6a0 msgr2=0x7fd6801018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 shutdown_connections 2024-05-26T12:02:48.911 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:48.910+0000 7fd68766a700 1 -- 172.21.15.156:0/3403275654 wait complete. 2024-05-26T12:02:49.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:49 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3403275654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:02:49.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:49 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3403275654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:02:49.604 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:02:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:49 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3403275654' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:02:49.849 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:50.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:50 smithi156 ceph-mon[32025]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:50.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:50 smithi192 ceph-mon[36545]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:50.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:50 smithi186 ceph-mon[36412]: pgmap v364: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-05-26T12:02:51.584 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 -- 172.21.15.156:0/3402785601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e301024f0 msgr2=0x7f1e301103d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 --2- 172.21.15.156:0/3402785601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e301024f0 0x7f1e301103d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f1e2800b0d0 tx=0x7f1e2800b3e0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 -- 172.21.15.156:0/3402785601 shutdown_connections 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 --2- 172.21.15.156:0/3402785601 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e301024f0 0x7f1e301103d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 --2- 172.21.15.156:0/3402785601 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e30109520 0x7f1e30109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 --2- 172.21.15.156:0/3402785601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30101b30 0x7f1e30101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 -- 172.21.15.156:0/3402785601 >> 172.21.15.156:0/3402785601 conn(0x7f1e30078ab0 msgr2=0x7f1e300ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 -- 172.21.15.156:0/3402785601 shutdown_connections 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.581+0000 7f1e3568c700 1 -- 172.21.15.156:0/3402785601 wait complete. 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 Processor -- start 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 -- start start 2024-05-26T12:02:51.585 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e301024f0 0x7f1e30103250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30109520 0x7f1e30103790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e30113760 con 0x7f1e30101b30 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e301135e0 con 0x7f1e30109520 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e3568c700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1e301138e0 con 0x7f1e301024f0 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51846/0 (socket says 172.21.15.156:51846) 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 -- 172.21.15.156:0/198784482 learned_addr learned my addr 172.21.15.156:0/198784482 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2e7fc700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e301024f0 0x7f1e30103250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2f7fe700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30109520 0x7f1e30103790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 -- 172.21.15.156:0/198784482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e301024f0 msgr2=0x7f1e30103250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e301024f0 0x7f1e30103250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.586 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 -- 172.21.15.156:0/198784482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30109520 msgr2=0x7f1e30103790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30109520 0x7f1e30103790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.582+0000 7f1e2effd700 1 -- 172.21.15.156:0/198784482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e301a80e0 con 0x7f1e30101b30 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e2effd700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000e9c0 tx=0x7f1e2000ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e2000cbd0 con 0x7f1e30101b30 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e301a83d0 con 0x7f1e30101b30 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e2000ee60 con 0x7f1e30101b30 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e200055c0 con 0x7f1e30101b30 2024-05-26T12:02:51.587 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.583+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e301a8910 con 0x7f1e30101b30 2024-05-26T12:02:51.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.584+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e30104e50 con 0x7f1e30101b30 2024-05-26T12:02:51.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.585+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f1e20004750 con 0x7f1e30101b30 2024-05-26T12:02:51.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.586+0000 7f1e17fff700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 0x7f1e180b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:51.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.586+0000 7f1e2e7fc700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 0x7f1e180b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:51.588 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.587+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f1e200e2720 con 0x7f1e30101b30 2024-05-26T12:02:51.589 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.587+0000 7f1e2e7fc700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 0x7f1e180b4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1e30108c10 tx=0x7f1e240073d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:51.590 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.589+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1e20054df0 con 0x7f1e30101b30 2024-05-26T12:02:51.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.845+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1e30104240 con 0x7f1e30101b30 2024-05-26T12:02:51.846 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.845+0000 7f1e17fff700 1 -- 172.21.15.156:0/198784482 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7f1e200aa3b0 con 0x7f1e30101b30 2024-05-26T12:02:51.846 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:02:51.846 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:02:51.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 msgr2=0x7f1e180b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:51.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 0x7f1e180b4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1e30108c10 tx=0x7f1e240073d0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 msgr2=0x7f1e30102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:51.848 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000e9c0 tx=0x7f1e2000ecd0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 shutdown_connections 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e180b2880 0x7f1e180b4d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e30109520 0x7f1e30103790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e30101b30 0x7f1e30102d10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 --2- 172.21.15.156:0/198784482 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e301024f0 0x7f1e30103250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 >> 172.21.15.156:0/198784482 conn(0x7f1e30078ab0 msgr2=0x7f1e30100290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 shutdown_connections 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:51.848+0000 7f1e3568c700 1 -- 172.21.15.156:0/198784482 wait complete. 2024-05-26T12:02:51.849 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:02:52.468 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:52 smithi156 ceph-mon[32025]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 12 op/s 2024-05-26T12:02:52.468 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:52 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/198784482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:52.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-26T12:00:34.689787+0000', 'modified': '2024-05-26T12:00:55.501071+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24476, 'mds_1': 24458, 'mds_2': 24449, 'mds_3': 24442, 'mds_4': 24431}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14601': {'gid': 14601, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6839/2826647780', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6838', 'nonce': 2826647780}, {'type': 'v1', 'addr': '172.21.15.156:6839', 'nonce': 2826647780}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14613': {'gid': 14613, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.156:6841/1148384159', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.156:6840', 'nonce': 1148384159}, {'type': 'v1', 'addr': '172.21.15.156:6841', 'nonce': 1148384159}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24397': {'gid': 24397, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6833/4217003952', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6832', 'nonce': 4217003952}, {'type': 'v1', 'addr': '172.21.15.192:6833', 'nonce': 4217003952}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24418': {'gid': 24418, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6835/699682135', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6834', 'nonce': 699682135}, {'type': 'v1', 'addr': '172.21.15.192:6835', 'nonce': 699682135}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24427': {'gid': 24427, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.192:6837/977346748', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6836', 'nonce': 977346748}, {'type': 'v1', 'addr': '172.21.15.192:6837', 'nonce': 977346748}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24431': {'gid': 24431, 'name': 'k', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 18, 'addr': '172.21.15.186:6833/586672120', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 586672120}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 586672120}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24442': {'gid': 24442, 'name': 'i', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 12, 'addr': '172.21.15.192:6839/1614885810', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.192:6838', 'nonce': 1614885810}, {'type': 'v1', 'addr': '172.21.15.192:6839', 'nonce': 1614885810}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24449': {'gid': 24449, 'name': 'e', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.186:6835/1012390034', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 1012390034}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 1012390034}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'h', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.186:6837/3519203877', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6836', 'nonce': 3519203877}, {'type': 'v1', 'addr': '172.21.15.186:6837', 'nonce': 3519203877}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24476': {'gid': 24476, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.186:6839/2121675127', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6838', 'nonce': 2121675127}, {'type': 'v1', 'addr': '172.21.15.186:6839', 'nonce': 2121675127}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24476, 'qdb_cluster': [24458, 24476, 24431, 24442, 24449]} 2024-05-26T12:02:52.496 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-26T12:02:52.496 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell mds.k status 2024-05-26T12:02:52.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:52 smithi192 ceph-mon[36545]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 12 op/s 2024-05-26T12:02:52.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:52 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/198784482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:52.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:52 smithi186 ceph-mon[36412]: pgmap v365: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 12 op/s 2024-05-26T12:02:52.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:52 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/198784482' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:02:52.742 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 -- 172.21.15.156:0/1156651448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c102e00 msgr2=0x7f1e5c10f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/1156651448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c102e00 0x7f1e5c10f2f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400b0a0 tx=0x7f1e5400b3b0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 -- 172.21.15.156:0/1156651448 shutdown_connections 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/1156651448 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c102e00 0x7f1e5c10f2f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/1156651448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c108520 0x7f1e5c108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/1156651448 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102440 0x7f1e5c1028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 -- 172.21.15.156:0/1156651448 >> 172.21.15.156:0/1156651448 conn(0x7f1e5c0793f0 msgr2=0x7f1e5c079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 -- 172.21.15.156:0/1156651448 shutdown_connections 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.392+0000 7f1e60fe1700 1 -- 172.21.15.156:0/1156651448 wait complete. 2024-05-26T12:02:54.395 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 Processor -- start 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 -- start start 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 0x7f1e5c19aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102e00 0x7f1e5c19afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c112670 con 0x7f1e5c108520 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1124f0 con 0x7f1e5c102440 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e60fe1700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c1127f0 con 0x7f1e5c102e00 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51870/0 (socket says 172.21.15.156:51870) 2024-05-26T12:02:54.396 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3793692558 learned_addr learned my addr 172.21.15.156:0/3793692558 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102e00 0x7f1e5c19afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.393+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 0x7f1e5c19aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3793692558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102e00 msgr2=0x7f1e5c19afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102e00 0x7f1e5c19afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3793692558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 msgr2=0x7f1e5c19aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 0x7f1e5c19aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3793692558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c1a6050 con 0x7f1e5c108520 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 0x7f1e5c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400b830 tx=0x7f1e54004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e4b7fe700 1 -- 172.21.15.156:0/3793692558 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54021070 con 0x7f1e5c108520 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c1a6220 con 0x7f1e5c108520 2024-05-26T12:02:54.397 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e4b7fe700 1 -- 172.21.15.156:0/3793692558 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e54007e30 con 0x7f1e5c108520 2024-05-26T12:02:54.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e4b7fe700 1 -- 172.21.15.156:0/3793692558 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e540047d0 con 0x7f1e5c108520 2024-05-26T12:02:54.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.394+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1a6680 con 0x7f1e5c108520 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.396+0000 7f1e4b7fe700 1 -- 172.21.15.156:0/3793692558 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f1e54007980 con 0x7f1e5c108520 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.397+0000 7f1e4b7fe700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 0x7f1e440b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.397+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 0x7f1e440b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.397+0000 7f1e4b7fe700 1 -- 172.21.15.156:0/3793692558 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f1e540e3340 con 0x7f1e5c108520 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.397+0000 7f1e6b877b80 1 Processor -- start 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 0x7f1e440b4d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0099d0 tx=0x7f1e4c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.400 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 -- start start 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f23798bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f2379c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f236f7c40 con 0x563f2379cf10 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f236e77e0 con 0x563f23798810 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f236eb820 con 0x563f237991c0 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51882/0 (socket says 172.21.15.156:51882) 2024-05-26T12:02:54.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5a59c700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f23798bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4036748533 learned_addr learned my addr 172.21.15.156:0/4036748533 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f2379c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4036748533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 msgr2=0x563f2379c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f2379c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4036748533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 msgr2=0x563f23798bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f23798bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4036748533 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f23637880 con 0x563f2379cf10 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.398+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f2379c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e540a7c30 tx=0x7f1e540099e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae279f529e24be2 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e48ff9700 1 -- 172.21.15.156:0/4036748533 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5402a070 con 0x563f2379cf10 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e48ff9700 1 -- 172.21.15.156:0/4036748533 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e540ab0c0 con 0x563f2379cf10 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e48ff9700 1 -- 172.21.15.156:0/4036748533 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54003c00 con 0x563f2379cf10 2024-05-26T12:02:54.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4036748533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 msgr2=0x563f2379d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.399+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e540a7c30 tx=0x7f1e540099e0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4036748533 shutdown_connections 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f2379d350 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f23798bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4036748533 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f2379c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4036748533 >> 172.21.15.156:0/4036748533 conn(0x563f236c6030 msgr2=0x563f23792e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4036748533 shutdown_connections 2024-05-26T12:02:54.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4036748533 wait complete. 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- start start 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 Processor -- start 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f2379cf10 0x563f237a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.406 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f235edf50 con 0x563f23798810 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f23689770 con 0x563f237991c0 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f237a4e90 con 0x563f2379cf10 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e6b877b80 10 client.0 _ll_register_callbacks cb 0x563f236828d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e5a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e5a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51886/0 (socket says 172.21.15.156:51886) 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.400+0000 7f1e5a59c700 1 -- 172.21.15.156:0/3454346559 learned_addr learned my addr 172.21.15.156:0/3454346559 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f2379cf10 0x563f237a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 -- 172.21.15.156:0/3454346559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f2379cf10 msgr2=0x563f237a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f2379cf10 0x563f237a1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 -- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 msgr2=0x563f237a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 -- 172.21.15.156:0/3454346559 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f23686640 con 0x563f23798810 2024-05-26T12:02:54.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00dea0 tx=0x7f1e4c004860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89e502170c46b8a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e41ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c024020 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f1e4c00f040 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e540ab0c0 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e4c02f050 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e6b877b80 10 client.14961 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f237af570 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.401+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e540ab0c0 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e4c013070 con 0x563f23798810 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 10 client.14961 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 10 client.14961 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 20 client.14961 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f237b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.408 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 4 client.14961 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 --> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f23764990 con 0x563f237b1c10 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f237b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e41ffb700 10 client.14961 ms_handle_connect on v2:172.21.15.186:6832/586672120 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.402+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f237b3080 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=cb60f37ee6a3c0d1 server_cookie=75bdc4a5a47d495b in_seq=0 out_seq=0 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e41ffb700 1 -- 172.21.15.156:0/3454346559 <== mds.4 v2:172.21.15.186:6832/586672120 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x563f23764990 con 0x563f237b1c10 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e41ffb700 10 client.14961 handle_command_reply: tid=0 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 client.14961 shutdown 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 msgr2=0x563f237a43a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00dea0 tx=0x7f1e4c004860 comp rx=0 tx=0).stop 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 shutdown_connections 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f237b3080 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f2379cf10 0x563f237a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e41ffb700 0 client.14961 ms_handle_reset on v2:172.21.15.186:6832/586672120 2024-05-26T12:02:54.409 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f23798810 0x563f237a43a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3454346559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.403+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 >> 172.21.15.156:0/3454346559 conn(0x563f236c6030 msgr2=0x563f237924c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.404+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 shutdown_connections 2024-05-26T12:02:54.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.404+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3454346559 wait complete. 2024-05-26T12:02:54.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.404+0000 7f1e6b877b80 20 client.14961 trim_cache size 0 max 16384 2024-05-26T12:02:54.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.424+0000 7f1e6b877b80 1 Processor -- start 2024-05-26T12:02:54.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.424+0000 7f1e6b877b80 1 -- start start 2024-05-26T12:02:54.425 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:54 smithi156 ceph-mon[32025]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f2379bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f23792c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f237b13c0 con 0x563f2379cf10 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f237b1a90 con 0x563f23798810 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f237b2160 con 0x563f237991c0 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51896/0 (socket says 172.21.15.156:51896) 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3573580443 learned_addr learned my addr 172.21.15.156:0/3573580443 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e59d9b700 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f23792c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f2379bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3573580443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 msgr2=0x563f23792c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f23792c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3573580443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 msgr2=0x563f2379bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f2379bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/3573580443 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f237b3fe0 con 0x563f2379cf10 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009fd0 tx=0x7f1e540b1040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58d7f6f631cee87c server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e43fff700 1 -- 172.21.15.156:0/3573580443 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e540c4070 con 0x563f2379cf10 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e43fff700 1 -- 172.21.15.156:0/3573580443 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e540aa9c0 con 0x563f2379cf10 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.425+0000 7f1e43fff700 1 -- 172.21.15.156:0/3573580443 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5402a040 con 0x563f2379cf10 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3573580443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 msgr2=0x563f23793180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009fd0 tx=0x7f1e540b1040 comp rx=0 tx=0).stop 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3573580443 shutdown_connections 2024-05-26T12:02:54.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f23793180 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f23798810 0x563f2379bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/3573580443 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f237991c0 0x563f23792c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3573580443 >> 172.21.15.156:0/3573580443 conn(0x563f236c6030 msgr2=0x563f2379b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3573580443 shutdown_connections 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- 172.21.15.156:0/3573580443 wait complete. 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.426+0000 7f1e6b877b80 1 -- start start 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 Processor -- start 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f23798810 0x563f237bcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237b8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x563f237b2f00 con 0x563f2379cf10 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x563f237b2830 con 0x563f237991c0 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x563f237b35d0 con 0x563f23798810 2024-05-26T12:02:54.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e6b877b80 10 client.0 _ll_register_callbacks cb 0x563f236828d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51904/0 (socket says 172.21.15.156:51904) 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e59d9b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237b8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4247812840 learned_addr learned my addr 172.21.15.156:0/4247812840 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f23798810 0x563f237bcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4247812840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f23798810 msgr2=0x563f237bcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f23798810 0x563f237bcc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 msgr2=0x563f237b8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237b8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 -- 172.21.15.156:0/4247812840 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f237b8e00 con 0x563f2379cf10 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e5ad9d700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1e540a6a40 tx=0x7f1e540a6880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1905a4691efe563 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.427+0000 7f1e41ffb700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:54.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5406f060 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f1e540074e0 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563f236f9b90 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1e540c4030 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 10 client.14967 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563f2381d530 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x563f236f9b90 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1e54011070 con 0x563f2379cf10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 10 client.14967 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 10 client.14967 resolve_mds: resolved name 'k' to daemon mds.k 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 20 client.14967 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f23820940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 4 client.14967 mds_command: new command op to 24431 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 --> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] -- command(tid 0: {"prefix": "status"}) -- 0x563f236f3370 con 0x563f237b1c10 2024-05-26T12:02:54.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.428+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f23820940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e5a59c700 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f23820940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=449c20acb9db0456 server_cookie=3b0bda32a215eb9c in_seq=0 out_seq=0 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e41ffb700 10 client.14967 ms_handle_connect on v2:172.21.15.186:6832/586672120 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e41ffb700 1 -- 172.21.15.156:0/4247812840 <== mds.4 v2:172.21.15.186:6832/586672120 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x563f236f3370 con 0x563f237b1c10 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e41ffb700 10 client.14967 handle_command_reply: tid=0 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 client.14967 shutdown 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 msgr2=0x563f237b85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1e540a6a40 tx=0x7f1e540a6880 comp rx=0 tx=0).stop 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 shutdown_connections 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] conn(0x563f237b1c10 0x563f23820940 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x563f2379cf10 0x563f237b85c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.435 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e41ffb700 0 client.14967 ms_handle_reset on v2:172.21.15.186:6832/586672120 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x563f23798810 0x563f237bcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 --2- 172.21.15.156:0/4247812840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x563f237991c0 0x563f237b8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 >> 172.21.15.156:0/4247812840 conn(0x563f236c6030 msgr2=0x563f2379c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 shutdown_connections 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 1 -- 172.21.15.156:0/4247812840 wait complete. 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.429+0000 7f1e6b877b80 20 client.14967 trim_cache size 0 max 16384 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 4, 2024-05-26T12:02:54.436 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 24431, 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 122.758038137, 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 27, 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 83, 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 79, 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 189.07174589600001 2024-05-26T12:02:54.437 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 msgr2=0x7f1e440b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 0x7f1e440b4d40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c0099d0 tx=0x7f1e4c008040 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 msgr2=0x7f1e5c1a5b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400b830 tx=0x7f1e54004ce0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 shutdown_connections 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1e440b2880 0x7f1e440b4d40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1e5c108520 0x7f1e5c1a5b10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e5c102440 0x7f1e5c19aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 --2- 172.21.15.156:0/3793692558 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1e5c102e00 0x7f1e5c19afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 >> 172.21.15.156:0/3793692558 conn(0x7f1e5c0793f0 msgr2=0x7f1e5c100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:54.438 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 shutdown_connections 2024-05-26T12:02:54.439 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:54.430+0000 7f1e60fe1700 1 -- 172.21.15.156:0/3793692558 wait complete. 2024-05-26T12:02:54.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:54 smithi192 ceph-mon[36545]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:02:54.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:54 smithi186 ceph-mon[36412]: pgmap v366: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:02:55.108 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell mds.i status 2024-05-26T12:02:55.351 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:56.531 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:56 smithi156 ceph-mon[32025]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:56.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:56 smithi192 ceph-mon[36545]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:56.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:56 smithi186 ceph-mon[36412]: pgmap v367: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:02:57.089 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 -- 172.21.15.156:0/4163649173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 msgr2=0x7fd398109990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 --2- 172.21.15.156:0/4163649173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398109990 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd388009990 tx=0x7fd388009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 -- 172.21.15.156:0/4163649173 shutdown_connections 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 --2- 172.21.15.156:0/4163649173 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806bbe0 0x7fd3981103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 --2- 172.21.15.156:0/4163649173 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398109990 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 --2- 172.21.15.156:0/4163649173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806b220 0x7fd39806b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 -- 172.21.15.156:0/4163649173 >> 172.21.15.156:0/4163649173 conn(0x7fd3980793f0 msgr2=0x7fd398079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 -- 172.21.15.156:0/4163649173 shutdown_connections 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.088+0000 7fd39d452700 1 -- 172.21.15.156:0/4163649173 wait complete. 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.089+0000 7fd39d452700 1 Processor -- start 2024-05-26T12:02:57.090 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.089+0000 7fd39d452700 1 -- start start 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.089+0000 7fd39d452700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806b220 0x7fd39819f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.089+0000 7fd39d452700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 0x7fd39819fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.089+0000 7fd39d452700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd39d452700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fd398113740 con 0x7fd3981095b0 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd39d452700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981135c0 con 0x7fd39806bbe0 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd39d452700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fd3981138c0 con 0x7fd39806b220 2024-05-26T12:02:57.092 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd396ffd700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806b220 0x7fd39819f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51780/0 (socket says 172.21.15.156:51780) 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 -- 172.21.15.156:0/1727617900 learned_addr learned my addr 172.21.15.156:0/1727617900 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3967fc700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 0x7fd39819fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 -- 172.21.15.156:0/1727617900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806b220 msgr2=0x7fd39819f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806b220 0x7fd39819f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 -- 172.21.15.156:0/1727617900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 msgr2=0x7fd39819fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 0x7fd39819fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 -- 172.21.15.156:0/1727617900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd39819a220 con 0x7fd3981095b0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3967fc700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 0x7fd39819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd3977fe700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd39000b970 tx=0x7fd39000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd37ffff700 1 -- 172.21.15.156:0/1727617900 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390019070 con 0x7fd3981095b0 2024-05-26T12:02:57.093 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd39819a510 con 0x7fd3981095b0 2024-05-26T12:02:57.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd37ffff700 1 -- 172.21.15.156:0/1727617900 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd39000cde0 con 0x7fd3981095b0 2024-05-26T12:02:57.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd37ffff700 1 -- 172.21.15.156:0/1727617900 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd390007370 con 0x7fd3981095b0 2024-05-26T12:02:57.094 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.090+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3981a6780 con 0x7fd3981095b0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.093+0000 7fd37ffff700 1 -- 172.21.15.156:0/1727617900 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fd39001c070 con 0x7fd3981095b0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd37ffff700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 0x7fd3800b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd396ffd700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 0x7fd3800b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd37ffff700 1 -- 172.21.15.156:0/1727617900 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fd3900e1770 con 0x7fd3981095b0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 Processor -- start 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 -- start start 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd396ffd700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 0x7fd3800b4d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd38800b5c0 tx=0x7fd388005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad80ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad813350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ebad76dc40 con 0x55ebad80f1c0 2024-05-26T12:02:57.096 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebad75d7e0 con 0x55ebad812f10 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ebad761820 con 0x55ebad80e810 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.094+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51782/0 (socket says 172.21.15.156:51782) 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 -- 172.21.15.156:0/3000624855 learned_addr learned my addr 172.21.15.156:0/3000624855 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd396ffd700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad80ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3977fe700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad813350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 -- 172.21.15.156:0/3000624855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 msgr2=0x55ebad80ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad80ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 -- 172.21.15.156:0/3000624855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 msgr2=0x55ebad813350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.097 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad813350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 -- 172.21.15.156:0/3000624855 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebad6ad880 con 0x55ebad80f1c0 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd396ffd700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad80ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3977fe700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad813350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd38c008c50 tx=0x7fd38c008f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff0153dc644feaf2 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd37d7fa700 1 -- 172.21.15.156:0/3000624855 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c014070 con 0x55ebad80f1c0 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd37d7fa700 1 -- 172.21.15.156:0/3000624855 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38c00edf0 con 0x55ebad80f1c0 2024-05-26T12:02:57.098 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.095+0000 7fd37d7fa700 1 -- 172.21.15.156:0/3000624855 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c010430 con 0x55ebad80f1c0 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3000624855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 msgr2=0x55ebad8128f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd38c008c50 tx=0x7fd38c008f60 comp rx=0 tx=0).stop 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3000624855 shutdown_connections 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad813350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad80ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3000624855 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad8128f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3000624855 >> 172.21.15.156:0/3000624855 conn(0x55ebad73c030 msgr2=0x55ebad808e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.102 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3000624855 shutdown_connections 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3000624855 wait complete. 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.096+0000 7fd3a7ce8b80 1 -- start start 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 Processor -- start 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad81a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad817ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ebad663f50 con 0x55ebad80f1c0 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebad6ff770 con 0x55ebad812f10 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ebad81b0d0 con 0x55ebad80e810 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3a7ce8b80 10 client.0 _ll_register_callbacks cb 0x55ebad6f88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51798/0 (socket says 172.21.15.156:51798) 2024-05-26T12:02:57.103 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 -- 172.21.15.156:0/3897828703 learned_addr learned my addr 172.21.15.156:0/3897828703 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd396ffd700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad81a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3977fe700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad817ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 -- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 msgr2=0x55ebad81a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad81a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 -- 172.21.15.156:0/3897828703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 msgr2=0x55ebad817ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad817ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 -- 172.21.15.156:0/3897828703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebad6fc640 con 0x55ebad80f1c0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd3967fc700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005230 tx=0x7fd38c005260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b18b5cdc22d02ee6 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd37aff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.097+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c014070 con 0x55ebad80f1c0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fd38c004c80 con 0x55ebad80f1c0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd38c00edf0 con 0x55ebad80f1c0 2024-05-26T12:02:57.104 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd38c004020 con 0x55ebad80f1c0 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 10 client.14979 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebad825680 con 0x55ebad80f1c0 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38c00edf0 con 0x55ebad80f1c0 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd38c02c070 con 0x55ebad80f1c0 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 10 client.14979 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 10 client.14979 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 20 client.14979 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad827860 0x55ebad828cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 4 client.14979 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 --> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebad7da990 con 0x55ebad827860 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.098+0000 7fd396ffd700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad827860 0x55ebad828cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.105 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.099+0000 7fd37aff5700 10 client.14979 ms_handle_connect on v2:172.21.15.192:6838/1614885810 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.099+0000 7fd396ffd700 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad827860 0x55ebad828cd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=95016cba4d0f41cc server_cookie=e83889490e9bca70 in_seq=0 out_seq=0 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd37aff5700 1 -- 172.21.15.156:0/3897828703 <== mds.3 v2:172.21.15.192:6838/1614885810 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x55ebad7da990 con 0x55ebad827860 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd37aff5700 10 client.14979 handle_command_reply: tid=0 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 client.14979 shutdown 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 msgr2=0x55ebad817590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005230 tx=0x7fd38c005260 comp rx=0 tx=0).stop 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 shutdown_connections 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad827860 0x55ebad828cd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad817ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad81a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/3897828703 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad817590 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 >> 172.21.15.156:0/3897828703 conn(0x55ebad73c030 msgr2=0x55ebad8084c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.106 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd37aff5700 0 client.14979 ms_handle_reset on v2:172.21.15.192:6838/1614885810 2024-05-26T12:02:57.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 shutdown_connections 2024-05-26T12:02:57.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/3897828703 wait complete. 2024-05-26T12:02:57.107 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.100+0000 7fd3a7ce8b80 20 client.14979 trim_cache size 0 max 16384 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 Processor -- start 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 -- start start 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad8164f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad832520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ebad827010 con 0x55ebad80f1c0 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebad8276e0 con 0x55ebad812f10 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ebad827db0 con 0x55ebad80e810 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3967fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51810/0 (socket says 172.21.15.156:51810) 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.121+0000 7fd3967fc700 1 -- 172.21.15.156:0/886672317 learned_addr learned my addr 172.21.15.156:0/886672317 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:57.127 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3977fe700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad832520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd396ffd700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad8164f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 -- 172.21.15.156:0/886672317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 msgr2=0x55ebad8164f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad8164f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 -- 172.21.15.156:0/886672317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 msgr2=0x55ebad832520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad832520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 -- 172.21.15.156:0/886672317 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebad826df0 con 0x55ebad80f1c0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd396ffd700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad8164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd3967fc700 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005950 tx=0x7fd38c008f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6174d490f4c80aa server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd37cff9700 1 -- 172.21.15.156:0/886672317 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c033070 con 0x55ebad80f1c0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd37cff9700 1 -- 172.21.15.156:0/886672317 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38c029070 con 0x55ebad80f1c0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.122+0000 7fd37cff9700 1 -- 172.21.15.156:0/886672317 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c014070 con 0x55ebad80f1c0 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/886672317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 msgr2=0x55ebad808c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005950 tx=0x7fd38c008f80 comp rx=0 tx=0).stop 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/886672317 shutdown_connections 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad812f10 0x55ebad832520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad8164f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/886672317 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad80f1c0 0x55ebad808c40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/886672317 >> 172.21.15.156:0/886672317 conn(0x55ebad73c030 msgr2=0x55ebad80deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/886672317 shutdown_connections 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/886672317 wait complete. 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- start start 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 Processor -- start 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad830770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad80f1c0 0x55ebad830cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x55ebad828b50 con 0x55ebad812f10 2024-05-26T12:02:57.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x55ebad828480 con 0x55ebad80f1c0 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.123+0000 7fd3a7ce8b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x55ebad829220 con 0x55ebad80e810 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3a7ce8b80 10 client.0 _ll_register_callbacks cb 0x55ebad6f88d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51812/0 (socket says 172.21.15.156:51812) 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 -- 172.21.15.156:0/407435904 learned_addr learned my addr 172.21.15.156:0/407435904 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd396ffd700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad830770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3967fc700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad80f1c0 0x55ebad830cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 -- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 msgr2=0x55ebad830770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad830770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 -- 172.21.15.156:0/407435904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad80f1c0 msgr2=0x55ebad830cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad80f1c0 0x55ebad830cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 -- 172.21.15.156:0/407435904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebad8907a0 con 0x55ebad812f10 2024-05-26T12:02:57.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3977fe700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd3900a8cf0 tx=0x7fd390004be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa5a6f5f85388b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd37aff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3900c3430 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fd3900b1940 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd38c029070 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fd3900ad110 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3a7ce8b80 10 client.14985 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ebad897b20 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.124+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38c029070 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7fd3900a4860 con 0x55ebad812f10 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 10 client.14985 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:57.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 10 client.14985 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 20 client.14985 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad89c760 0x55ebad89b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 4 client.14985 mds_command: new command op to 24442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 --> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] -- command(tid 0: {"prefix": "status"}) -- 0x55ebad76fb90 con 0x55ebad89c760 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd396ffd700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad89c760 0x55ebad89b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd37aff5700 10 client.14985 ms_handle_connect on v2:172.21.15.192:6838/1614885810 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.125+0000 7fd396ffd700 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad89c760 0x55ebad89b470 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=462fec56cfcba18 server_cookie=349b92df51218c5e in_seq=0 out_seq=0 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd37aff5700 1 -- 172.21.15.156:0/407435904 <== mds.3 v2:172.21.15.192:6838/1614885810 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ebad76fb90 con 0x55ebad89c760 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd37aff5700 10 client.14985 handle_command_reply: tid=0 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 client.14985 shutdown 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 msgr2=0x55ebad890160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd3900a8cf0 tx=0x7fd390004be0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 shutdown_connections 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] conn(0x55ebad89c760 0x55ebad89b470 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x55ebad812f10 0x55ebad890160 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x55ebad80e810 0x55ebad830770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 --2- 172.21.15.156:0/407435904 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55ebad80f1c0 0x55ebad830cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 >> 172.21.15.156:0/407435904 conn(0x55ebad73c030 msgr2=0x55ebad811860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd37aff5700 0 client.14985 ms_handle_reset on v2:172.21.15.192:6838/1614885810 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 shutdown_connections 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 1 -- 172.21.15.156:0/407435904 wait complete. 2024-05-26T12:02:57.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.126+0000 7fd3a7ce8b80 20 client.14985 trim_cache size 0 max 16384 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 3, 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 24442, 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 126.464717509, 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 27, 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 83, 2024-05-26T12:02:57.134 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 79, 2024-05-26T12:02:57.135 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 170.28638110899999 2024-05-26T12:02:57.135 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T12:02:57.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 msgr2=0x7fd3800b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.135 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 0x7fd3800b4d40 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd38800b5c0 tx=0x7fd388005960 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 msgr2=0x7fd398199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd39000b970 tx=0x7fd39000bc80 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 shutdown_connections 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fd3800b2880 0x7fd3800b4d40 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fd3981095b0 0x7fd398199ac0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fd39806b220 0x7fd39819f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 --2- 172.21.15.156:0/1727617900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd39806bbe0 0x7fd39819fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 >> 172.21.15.156:0/1727617900 conn(0x7fd3980793f0 msgr2=0x7fd398101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 shutdown_connections 2024-05-26T12:02:57.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:57.127+0000 7fd39d452700 1 -- 172.21.15.156:0/1727617900 wait complete. 2024-05-26T12:02:57.906 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell mds.e status 2024-05-26T12:02:58.148 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:02:58.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:02:58 smithi156 ceph-mon[32025]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:58.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:02:58 smithi192 ceph-mon[36545]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:58.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:02:58 smithi186 ceph-mon[36412]: pgmap v368: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:02:59.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 -- 172.21.15.156:0/550049241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394108520 msgr2=0x7f1394108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 --2- 172.21.15.156:0/550049241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394108520 0x7f1394108900 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f1388009990 tx=0x7f1388009ca0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 -- 172.21.15.156:0/550049241 shutdown_connections 2024-05-26T12:02:59.855 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 --2- 172.21.15.156:0/550049241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394102e00 0x7f139410f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 --2- 172.21.15.156:0/550049241 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394108520 0x7f1394108900 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 --2- 172.21.15.156:0/550049241 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102440 0x7f13941028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 -- 172.21.15.156:0/550049241 >> 172.21.15.156:0/550049241 conn(0x7f13940793f0 msgr2=0x7f1394079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 -- 172.21.15.156:0/550049241 shutdown_connections 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.854+0000 7f13995ee700 1 -- 172.21.15.156:0/550049241 wait complete. 2024-05-26T12:02:59.856 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 Processor -- start 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 -- start start 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 0x7f139419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394108520 0x7f13941989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1394112650 con 0x7f1394102440 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f13941124d0 con 0x7f1394108520 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f13995ee700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f13941127d0 con 0x7f1394102e00 2024-05-26T12:02:59.858 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51830/0 (socket says 172.21.15.156:51830) 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.855+0000 7f1392ffd700 1 -- 172.21.15.156:0/3082297343 learned_addr learned my addr 172.21.15.156:0/3082297343 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f13937fe700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394108520 0x7f13941989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f13927fc700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 0x7f139419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 -- 172.21.15.156:0/3082297343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 msgr2=0x7f139419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 0x7f139419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 -- 172.21.15.156:0/3082297343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394108520 msgr2=0x7f13941989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394108520 0x7f13941989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 -- 172.21.15.156:0/3082297343 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1394199130 con 0x7f1394102440 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f13927fc700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 0x7f139419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1388000c00 tx=0x7f13880047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f137bfff700 1 -- 172.21.15.156:0/3082297343 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138802a070 con 0x7f1394102440 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f137bfff700 1 -- 172.21.15.156:0/3082297343 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f138801ab40 con 0x7f1394102440 2024-05-26T12:02:59.859 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13941993c0 con 0x7f1394102440 2024-05-26T12:02:59.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f137bfff700 1 -- 172.21.15.156:0/3082297343 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388004110 con 0x7f1394102440 2024-05-26T12:02:59.860 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.856+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f139406b220 con 0x7f1394102440 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.858+0000 7f137bfff700 1 -- 172.21.15.156:0/3082297343 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f138802d070 con 0x7f1394102440 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f137bfff700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 0x7f137c0b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f13927fc700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 0x7f137c0b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f137bfff700 1 -- 172.21.15.156:0/3082297343 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f13880e5160 con 0x7f1394102440 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f13927fc700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 0x7f137c0b4dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1394079ba0 tx=0x7f1384008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f13a3e84b80 1 Processor -- start 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.860+0000 7f13a3e84b80 1 -- start start 2024-05-26T12:02:59.862 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7bc60 0x559aeea7c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea88410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559aee9d5d20 con 0x559aeea81670 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559aee9c58c0 con 0x559aeea7bc60 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559aee9c9900 con 0x559aeea7c620 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51834/0 (socket says 172.21.15.156:51834) 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 -- 172.21.15.156:0/3719682371 learned_addr learned my addr 172.21.15.156:0/3719682371 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13927fc700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7bc60 0x559aeea7c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13937fe700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea88410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 -- 172.21.15.156:0/3719682371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 msgr2=0x559aeea88410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea88410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 -- 172.21.15.156:0/3719682371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7bc60 msgr2=0x559aeea7c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.863 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7bc60 0x559aeea7c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 -- 172.21.15.156:0/3719682371 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aee96ef60 con 0x559aeea81670 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13937fe700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea88410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13880a8860 tx=0x7f13880ac6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377c02f17f5b2a85 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.861+0000 7f13797fa700 1 -- 172.21.15.156:0/3719682371 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880cf0b0 con 0x559aeea81670 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13797fa700 1 -- 172.21.15.156:0/3719682371 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13880b30e0 con 0x559aeea81670 2024-05-26T12:02:59.864 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13797fa700 1 -- 172.21.15.156:0/3719682371 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388003860 con 0x559aeea81670 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3719682371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 msgr2=0x559aeea81a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f13880a8860 tx=0x7f13880ac6e0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3719682371 shutdown_connections 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea88410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea81670 0x559aeea81a50 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3719682371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7bc60 0x559aeea7c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3719682371 >> 172.21.15.156:0/3719682371 conn(0x559aee9a31f0 msgr2=0x559aeea72390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.868 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3719682371 shutdown_connections 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.862+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3719682371 wait complete. 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 -- start start 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 Processor -- start 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea85380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea858c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559aee8cbf50 con 0x559aeea7bc60 2024-05-26T12:02:59.869 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559aee92dd50 con 0x559aeea81670 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559aeea88950 con 0x559aeea7c620 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f13a3e84b80 10 client.0 _ll_register_callbacks cb 0x559aee960900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51844/0 (socket says 172.21.15.156:51844) 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.863+0000 7f1392ffd700 1 -- 172.21.15.156:0/2915848032 learned_addr learned my addr 172.21.15.156:0/2915848032 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13927fc700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea85380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13937fe700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea858c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 -- 172.21.15.156:0/2915848032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 msgr2=0x559aeea85380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea85380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 -- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 msgr2=0x559aeea858c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea858c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 -- 172.21.15.156:0/2915848032 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aee915880 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13937fe700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea858c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13880b30b0 tx=0x7f13880a87d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bc8d4342ccd8a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1376ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880b55b0 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f13880aca40 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13880b30e0 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f13880c6020 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13a3e84b80 10 client.14997 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559aee97f710 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.864+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13880b30e0 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f13880c1ba0 con 0x559aeea7bc60 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 10 client.14997 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:59.871 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 10 client.14997 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 20 client.14997 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeea950e0 0x559aeea96550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 4 client.14997 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 --> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559aee9d7c70 con 0x559aeea950e0 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13927fc700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeea950e0 0x559aeea96550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f13927fc700 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeea950e0 0x559aeea96550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f7d173ddd0d9e882 server_cookie=65fd7cf532392eae in_seq=0 out_seq=0 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.865+0000 7f1376ff5700 10 client.14997 ms_handle_connect on v2:172.21.15.186:6834/1012390034 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f1376ff5700 1 -- 172.21.15.156:0/2915848032 <== mds.2 v2:172.21.15.186:6834/1012390034 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x559aee9d7c70 con 0x559aeea950e0 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f1376ff5700 10 client.14997 handle_command_reply: tid=0 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 client.14997 shutdown 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 msgr2=0x559aeea88080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f13880b30b0 tx=0x7f13880a87d0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.872 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 shutdown_connections 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea858c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea88080 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeea950e0 0x559aeea96550 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2915848032 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7c620 0x559aeea85380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f1376ff5700 0 client.14997 ms_handle_reset on v2:172.21.15.186:6834/1012390034 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.866+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 >> 172.21.15.156:0/2915848032 conn(0x559aee9a31f0 msgr2=0x559aeea71aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.867+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 shutdown_connections 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.867+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2915848032 wait complete. 2024-05-26T12:02:59.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.867+0000 7f13a3e84b80 20 client.14997 trim_cache size 0 max 16384 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.889+0000 7f13a3e84b80 1 Processor -- start 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.889+0000 7f13a3e84b80 1 -- start start 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 0x559aeea72220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 0x559aeea76840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.891 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559aeea94890 con 0x559aeea7bc60 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559aeea94f60 con 0x559aeea7c620 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559aeea95630 con 0x559aeea81670 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51862/0 (socket says 172.21.15.156:51862) 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 -- 172.21.15.156:0/3439041306 learned_addr learned my addr 172.21.15.156:0/3439041306 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13927fc700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 0x559aeea72220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13937fe700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 0x559aeea76840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 -- 172.21.15.156:0/3439041306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 msgr2=0x559aeea76840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 0x559aeea76840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 -- 172.21.15.156:0/3439041306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 msgr2=0x559aeea72220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 0x559aeea72220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 -- 172.21.15.156:0/3439041306 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559aeea87cc0 con 0x559aeea7bc60 2024-05-26T12:02:59.892 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13927fc700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 0x559aeea72220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f13937fe700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 0x559aeea76840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:02:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1392ffd700 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13880ac770 tx=0x7f13880ac6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91b48d7f82d6818 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1378ff9700 1 -- 172.21.15.156:0/3439041306 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880cf0b0 con 0x559aeea7bc60 2024-05-26T12:02:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1378ff9700 1 -- 172.21.15.156:0/3439041306 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13880b55b0 con 0x559aeea7bc60 2024-05-26T12:02:59.893 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.890+0000 7f1378ff9700 1 -- 172.21.15.156:0/3439041306 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13880aca40 con 0x559aeea7bc60 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3439041306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 msgr2=0x559aeea7f7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f13880ac770 tx=0x7f13880ac6e0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3439041306 shutdown_connections 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea81670 0x559aeea76840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7bc60 0x559aeea7f7b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/3439041306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea7c620 0x559aeea72220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3439041306 >> 172.21.15.156:0/3439041306 conn(0x559aee9a31f0 msgr2=0x559aeea7f0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3439041306 shutdown_connections 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.891+0000 7f13a3e84b80 1 -- 172.21.15.156:0/3439041306 wait complete. 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 -- start start 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 Processor -- start 2024-05-26T12:02:59.897 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7bc60 0x559aeea9fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x559aeea963d0 con 0x559aeea7c620 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x559aeea95d00 con 0x559aeea81670 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x559aeea96aa0 con 0x559aeea7bc60 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13a3e84b80 10 client.0 _ll_register_callbacks cb 0x559aee960900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13927fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13927fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51864/0 (socket says 172.21.15.156:51864) 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13927fc700 1 -- 172.21.15.156:0/2232817291 learned_addr learned my addr 172.21.15.156:0/2232817291 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7bc60 0x559aeea9fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.892+0000 7f13937fe700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 -- 172.21.15.156:0/2232817291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7bc60 msgr2=0x559aeea9fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.898 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7bc60 0x559aeea9fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 -- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 msgr2=0x559aeea9aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea9aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 -- 172.21.15.156:0/2232817291 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559aeea9b220 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13927fc700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x559aeea74010 tx=0x7f1384010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea1a521523db782 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f1376ff5700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f138400de50 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f138400f040 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13880b55b0 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1384014cb0 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13a3e84b80 10 client.15003 fetch_fsmap learned FSMap version 27 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x559aeeb014e0 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.893+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f13880b55b0 con 0x559aeea7c620 2024-05-26T12:02:59.899 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f138403a070 con 0x559aeea7c620 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 10 client.15003 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 10 client.15003 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 20 client.15003 populate_metadata read hostname 'smithi156' 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeeb060e0 0x559aeeb04df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 4 client.15003 mds_command: new command op to 24449 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 --> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] -- command(tid 0: {"prefix": "status"}) -- 0x559aee9d1450 con 0x559aeeb060e0 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeeb060e0 0x559aeeb04df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f1392ffd700 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeeb060e0 0x559aeeb04df0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7f19db0878907493 server_cookie=f5d22fa09d24b42e in_seq=0 out_seq=0 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.894+0000 7f1376ff5700 10 client.15003 ms_handle_connect on v2:172.21.15.186:6834/1012390034 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f1376ff5700 1 -- 172.21.15.156:0/2232817291 <== mds.2 v2:172.21.15.186:6834/1012390034 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x559aee9d1450 con 0x559aeeb060e0 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f1376ff5700 10 client.15003 handle_command_reply: tid=0 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 client.15003 shutdown 2024-05-26T12:02:59.900 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 msgr2=0x559aeeaa01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x559aeea74010 tx=0x7f1384010000 comp rx=0 tx=0).stop 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 shutdown_connections 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x559aeea81670 0x559aeea9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] conn(0x559aeeb060e0 0x559aeeb04df0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f1376ff5700 0 client.15003 ms_handle_reset on v2:172.21.15.186:6834/1012390034 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x559aeea7bc60 0x559aeea9fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 --2- 172.21.15.156:0/2232817291 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x559aeea7c620 0x559aeeaa01d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 >> 172.21.15.156:0/2232817291 conn(0x559aee9a31f0 msgr2=0x559aeea76840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 shutdown_connections 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 1 -- 172.21.15.156:0/2232817291 wait complete. 2024-05-26T12:02:59.901 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.895+0000 7f13a3e84b80 20 client.15003 trim_cache size 0 max 16384 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 2, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 24449, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 130.24031204400001, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 27, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 83, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 79, 2024-05-26T12:02:59.902 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 184.21436121100001 2024-05-26T12:02:59.903 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T12:02:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 msgr2=0x7f137c0b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 0x7f137c0b4dc0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1394079ba0 tx=0x7f1384008040 comp rx=0 tx=0).stop 2024-05-26T12:02:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 msgr2=0x7f139419e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:02:59.903 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f1388000c00 tx=0x7f13880047b0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 shutdown_connections 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f137c0b2900 0x7f137c0b4dc0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1394108520 0x7f13941989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1394102440 0x7f139419e810 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 --2- 172.21.15.156:0/3082297343 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1394102e00 0x7f139419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 >> 172.21.15.156:0/3082297343 conn(0x7f13940793f0 msgr2=0x7f1394101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 shutdown_connections 2024-05-26T12:02:59.904 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:02:59.896+0000 7f13995ee700 1 -- 172.21.15.156:0/3082297343 wait complete. 2024-05-26T12:03:00.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:00 smithi156 ceph-mon[32025]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:00.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:00 smithi192 ceph-mon[36545]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:00.641 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell mds.h status 2024-05-26T12:03:00.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:00 smithi186 ceph-mon[36412]: pgmap v369: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:00.900 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:02.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:02 smithi156 ceph-mon[32025]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:02.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:02 smithi192 ceph-mon[36545]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:02.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:02 smithi186 ceph-mon[36412]: pgmap v370: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 -- 172.21.15.156:0/1204961750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1588108620 msgr2=0x7f1588108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 --2- 172.21.15.156:0/1204961750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1588108620 0x7f1588108a00 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f1570009990 tx=0x7f1570009ca0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 -- 172.21.15.156:0/1204961750 shutdown_connections 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 --2- 172.21.15.156:0/1204961750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15881028c0 0x7f1588110330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 --2- 172.21.15.156:0/1204961750 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1588108620 0x7f1588108a00 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 --2- 172.21.15.156:0/1204961750 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588101f00 0x7f1588102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 -- 172.21.15.156:0/1204961750 >> 172.21.15.156:0/1204961750 conn(0x7f1588077780 msgr2=0x7f1588077b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 -- 172.21.15.156:0/1204961750 shutdown_connections 2024-05-26T12:03:02.782 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.779+0000 7f158ef16700 1 -- 172.21.15.156:0/1204961750 wait complete. 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 Processor -- start 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 -- start start 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1588101f00 0x7f15881030e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588108620 0x7f1588103b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1588113760 con 0x7f15881028c0 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f15881135e0 con 0x7f1588101f00 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ef16700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f15881138e0 con 0x7f1588108620 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51878/0 (socket says 172.21.15.156:51878) 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 -- 172.21.15.156:0/1258072709 learned_addr learned my addr 172.21.15.156:0/1258072709 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158d4b3700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588108620 0x7f1588103b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.783 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1588101f00 0x7f15881030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 -- 172.21.15.156:0/1258072709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588108620 msgr2=0x7f1588103b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588108620 0x7f1588103b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 -- 172.21.15.156:0/1258072709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1588101f00 msgr2=0x7f15881030e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1588101f00 0x7f15881030e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.780+0000 7f1587fff700 1 -- 172.21.15.156:0/1258072709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15881a8100 con 0x7f15881028c0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f1587fff700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f157800ef10 tx=0x7f157800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f1585ffb700 1 -- 172.21.15.156:0/1258072709 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157800c9c0 con 0x7f15881028c0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15881a8390 con 0x7f15881028c0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f1585ffb700 1 -- 172.21.15.156:0/1258072709 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1578019070 con 0x7f15881028c0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f1585ffb700 1 -- 172.21.15.156:0/1258072709 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578014480 con 0x7f15881028c0 2024-05-26T12:03:02.784 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.781+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15881a88d0 con 0x7f15881028c0 2024-05-26T12:03:02.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.783+0000 7f1585ffb700 1 -- 172.21.15.156:0/1258072709 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f1578012050 con 0x7f15881028c0 2024-05-26T12:03:02.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.784+0000 7f1585ffb700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 0x7f15740b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.784+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 0x7f15740b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.786 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.784+0000 7f1585ffb700 1 -- 172.21.15.156:0/1258072709 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f15780e2190 con 0x7f15881028c0 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.784+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 0x7f15740b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f1570005960 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 Processor -- start 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 -- start start 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c02cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c03210 0x5560c4c0f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c4b60d20 con 0x5560c4c082d0 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5560c4b508c0 con 0x5560c4c02850 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f15997acb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560c4b54900 con 0x5560c4c03210 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51884/0 (socket says 172.21.15.156:51884) 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 -- 172.21.15.156:0/1993194761 learned_addr learned my addr 172.21.15.156:0/1993194761 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f1587fff700 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c02cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.787 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 -- 172.21.15.156:0/1993194761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c03210 msgr2=0x5560c4c0f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c03210 0x5560c4c0f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 -- 172.21.15.156:0/1993194761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 msgr2=0x5560c4c02cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c02cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.785+0000 7f158ccb2700 1 -- 172.21.15.156:0/1993194761 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c4af9f60 con 0x5560c4c082d0 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f158ccb2700 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1588078ce0 tx=0x7f157001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2b6346c4586405 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f157effd700 1 -- 172.21.15.156:0/1993194761 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157000be40 con 0x5560c4c082d0 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f157effd700 1 -- 172.21.15.156:0/1993194761 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1570003b30 con 0x5560c4c082d0 2024-05-26T12:03:02.788 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f157effd700 1 -- 172.21.15.156:0/1993194761 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157000fb00 con 0x5560c4c082d0 2024-05-26T12:03:02.791 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 -- 172.21.15.156:0/1993194761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 msgr2=0x5560c4c086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1588078ce0 tx=0x7f157001f000 comp rx=0 tx=0).stop 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 -- 172.21.15.156:0/1993194761 shutdown_connections 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c03210 0x5560c4c0f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c082d0 0x5560c4c086b0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 --2- 172.21.15.156:0/1993194761 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c02cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.786+0000 7f15997acb80 1 -- 172.21.15.156:0/1993194761 >> 172.21.15.156:0/1993194761 conn(0x5560c4b2e1f0 msgr2=0x5560c4bfd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- 172.21.15.156:0/1993194761 shutdown_connections 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- 172.21.15.156:0/1993194761 wait complete. 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- start start 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 Processor -- start 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.792 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c03210 0x5560c4c0bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c0c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c4a56f50 con 0x5560c4c02850 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5560c4ab8d50 con 0x5560c4c03210 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c0f630 con 0x5560c4c082d0 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f15997acb80 10 client.0 _ll_register_callbacks cb 0x5560c4aeb900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.787+0000 7f158ccb2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51898/0 (socket says 172.21.15.156:51898) 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 -- 172.21.15.156:0/678693498 learned_addr learned my addr 172.21.15.156:0/678693498 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f1587fff700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c03210 0x5560c4c0bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158d4b3700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c0c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 -- 172.21.15.156:0/678693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 msgr2=0x5560c4c0c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c0c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.793 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 -- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c03210 msgr2=0x5560c4c0bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c03210 0x5560c4c0bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 -- 172.21.15.156:0/678693498 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c4aa0880 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f158ccb2700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1570009ce0 tx=0x7f15700057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd85752493ad471 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f1567fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1570037070 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f157000f780 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1570003b30 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f1570043050 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f15997acb80 10 client.15015 fetch_fsmap learned FSMap version 27 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.788+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560c4b0a710 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1570003b30 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f1570026430 con 0x5560c4c02850 2024-05-26T12:03:02.794 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 10 client.15015 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 10 client.15015 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 20 client.15015 populate_metadata read hostname 'smithi156' 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c1aea0 0x5560c4c1cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 4 client.15015 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 --> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560c4b62c70 con 0x5560c4c1aea0 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.789+0000 7f1587fff700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c1aea0 0x5560c4c1cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.790+0000 7f1567fff700 10 client.15015 ms_handle_connect on v2:172.21.15.186:6836/3519203877 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.790+0000 7f1587fff700 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c1aea0 0x5560c4c1cfd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=821edbf7677d4b69 server_cookie=628e4073481b2794 in_seq=0 out_seq=0 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.790+0000 7f1567fff700 1 -- 172.21.15.156:0/678693498 <== mds.1 v2:172.21.15.186:6836/3519203877 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x5560c4b62c70 con 0x5560c4c1aea0 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.790+0000 7f1567fff700 10 client.15015 handle_command_reply: tid=0 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.790+0000 7f15997acb80 1 client.15015 shutdown 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 msgr2=0x5560c4c0ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.795 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1570009ce0 tx=0x7f15700057e0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 shutdown_connections 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c1aea0 0x5560c4c1cfd0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c0c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c02850 0x5560c4c0ecf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 --2- 172.21.15.156:0/678693498 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c03210 0x5560c4c0bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 >> 172.21.15.156:0/678693498 conn(0x5560c4b2e1f0 msgr2=0x5560c4bfcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f1567fff700 0 client.15015 ms_handle_reset on v2:172.21.15.186:6836/3519203877 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 shutdown_connections 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 1 -- 172.21.15.156:0/678693498 wait complete. 2024-05-26T12:03:02.796 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.791+0000 7f15997acb80 20 client.15015 trim_cache size 0 max 16384 2024-05-26T12:03:02.813 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 Processor -- start 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 -- start start 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c01840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c022c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1b310 con 0x5560c4c03210 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1b9e0 con 0x5560c4c02850 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f15997acb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1c0b0 con 0x5560c4c082d0 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51900/0 (socket says 172.21.15.156:51900) 2024-05-26T12:03:02.814 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 -- 172.21.15.156:0/3017418439 learned_addr learned my addr 172.21.15.156:0/3017418439 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f158ccb2700 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c01840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 -- 172.21.15.156:0/3017418439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 msgr2=0x5560c4c022c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c022c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 -- 172.21.15.156:0/3017418439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 msgr2=0x5560c4c01840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c01840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f1587fff700 1 -- 172.21.15.156:0/3017418439 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560c4c0e930 con 0x5560c4c03210 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.812+0000 7f158ccb2700 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c01840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f1587fff700 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f15780a9710 tx=0x7f15780dfb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59cbe4aa4af4bb7e server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f157e7fc700 1 -- 172.21.15.156:0/3017418439 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780cc0b0 con 0x5560c4c03210 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f157e7fc700 1 -- 172.21.15.156:0/3017418439 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15780a9e80 con 0x5560c4c03210 2024-05-26T12:03:02.815 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f157e7fc700 1 -- 172.21.15.156:0/3017418439 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780b2150 con 0x5560c4c03210 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f15997acb80 1 -- 172.21.15.156:0/3017418439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 msgr2=0x5560c4c01d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f15997acb80 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f15780a9710 tx=0x7f15780dfb30 comp rx=0 tx=0).stop 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.813+0000 7f15997acb80 1 -- 172.21.15.156:0/3017418439 shutdown_connections 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c082d0 0x5560c4c022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c02850 0x5560c4c01840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- 172.21.15.156:0/3017418439 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c01d80 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.819 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- 172.21.15.156:0/3017418439 >> 172.21.15.156:0/3017418439 conn(0x5560c4b2e1f0 msgr2=0x5560c4c06590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- 172.21.15.156:0/3017418439 shutdown_connections 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- 172.21.15.156:0/3017418439 wait complete. 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- start start 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 Processor -- start 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c02850 0x5560c4c20270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c082d0 0x5560c4c20cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1ce50 con 0x5560c4c03210 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1c780 con 0x5560c4c082d0 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x5560c4c1d520 con 0x5560c4c02850 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.814+0000 7f15997acb80 10 client.0 _ll_register_callbacks cb 0x5560c4aeb900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.820 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:51902/0 (socket says 172.21.15.156:51902) 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 -- 172.21.15.156:0/2666693643 learned_addr learned my addr 172.21.15.156:0/2666693643 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f158ccb2700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c02850 0x5560c4c20270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f158d4b3700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c082d0 0x5560c4c20cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 -- 172.21.15.156:0/2666693643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c02850 msgr2=0x5560c4c20270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c02850 0x5560c4c20270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 -- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c082d0 msgr2=0x5560c4c20cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c082d0 0x5560c4c20cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 -- 172.21.15.156:0/2666693643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560c4c24b70 con 0x5560c4c03210 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1587fff700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f15780dfcb0 tx=0x7f1578009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3d718b58dd5d138 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1567fff700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15780cc0b0 con 0x5560c4c03210 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f157800cd10 con 0x5560c4c03210 2024-05-26T12:03:02.821 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15780a9e80 con 0x5560c4c03210 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f15780c7020 con 0x5560c4c03210 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f15997acb80 10 client.15021 fetch_fsmap learned FSMap version 27 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.815+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5560c4c87820 con 0x5560c4c03210 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15780a9e80 con 0x5560c4c03210 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f15780b2150 con 0x5560c4c03210 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 10 client.15021 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 10 client.15021 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 20 client.15021 populate_metadata read hostname 'smithi156' 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c8be20 0x5560c4c8ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 4 client.15021 mds_command: new command op to 24458 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 --> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] -- command(tid 0: {"prefix": "status"}) -- 0x5560c4b5c450 con 0x5560c4c8be20 2024-05-26T12:03:02.822 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.816+0000 7f158ccb2700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c8be20 0x5560c4c8ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f158ccb2700 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c8be20 0x5560c4c8ab30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6ee67e490141961a server_cookie=c5a326839de1862 in_seq=0 out_seq=0 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f1567fff700 10 client.15021 ms_handle_connect on v2:172.21.15.186:6836/3519203877 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f1567fff700 1 -- 172.21.15.156:0/2666693643 <== mds.1 v2:172.21.15.186:6836/3519203877 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5560c4b5c450 con 0x5560c4c8be20 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f1567fff700 10 client.15021 handle_command_reply: tid=0 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 client.15021 shutdown 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 msgr2=0x5560c4c207b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f15780dfcb0 tx=0x7f1578009bd0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 shutdown_connections 2024-05-26T12:03:02.823 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] conn(0x5560c4c8be20 0x5560c4c8ab30 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5560c4c082d0 0x5560c4c20cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x5560c4c02850 0x5560c4c20270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 --2- 172.21.15.156:0/2666693643 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x5560c4c03210 0x5560c4c207b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f1567fff700 0 client.15021 ms_handle_reset on v2:172.21.15.186:6836/3519203877 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 >> 172.21.15.156:0/2666693643 conn(0x5560c4b2e1f0 msgr2=0x5560c4c01840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 shutdown_connections 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 1 -- 172.21.15.156:0/2666693643 wait complete. 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.817+0000 7f15997acb80 20 client.15021 trim_cache size 0 max 16384 2024-05-26T12:03:02.824 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 1, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 24458, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 134.16850671200001, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 27, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 83, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 79, 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 178.73878756900001 2024-05-26T12:03:02.825 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 msgr2=0x7f15740b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 0x7f15740b4d40 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f1570006010 tx=0x7f1570005960 comp rx=0 tx=0).stop 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 msgr2=0x7f1588103620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f157800ef10 tx=0x7f157800c3a0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 shutdown_connections 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15740b2880 0x7f15740b4d40 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1588108620 0x7f1588103b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1588101f00 0x7f15881030e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 --2- 172.21.15.156:0/1258072709 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f15881028c0 0x7f1588103620 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:02.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 >> 172.21.15.156:0/1258072709 conn(0x7f1588077780 msgr2=0x7f1588100b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:02.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 shutdown_connections 2024-05-26T12:03:02.827 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:02.818+0000 7f158ef16700 1 -- 172.21.15.156:0/1258072709 wait complete. 2024-05-26T12:03:03.477 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph tell mds.b status 2024-05-26T12:03:03.721 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:04.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:04 smithi156 ceph-mon[32025]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:04.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:04 smithi192 ceph-mon[36545]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:04.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:04 smithi186 ceph-mon[36412]: pgmap v371: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 -- 172.21.15.156:0/2644908958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 msgr2=0x7f04a010f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2644908958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a010f2f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f049800b0a0 tx=0x7f049800b3b0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 -- 172.21.15.156:0/2644908958 shutdown_connections 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2644908958 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a010f2f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2644908958 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a01084b0 0x7f04a0108890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.414 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2644908958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a0102470 0x7f04a01028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 -- 172.21.15.156:0/2644908958 >> 172.21.15.156:0/2644908958 conn(0x7f04a0079410 msgr2=0x7f04a0079820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 -- 172.21.15.156:0/2644908958 shutdown_connections 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.406+0000 7f04a68a6700 1 -- 172.21.15.156:0/2644908958 wait complete. 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 Processor -- start 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 -- start start 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a0102470 0x7f04a019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 0x7f04a0198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f04a01125e0 con 0x7f04a0102e30 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f04a0112460 con 0x7f04a01084b0 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a68a6700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f04a0112760 con 0x7f04a0102470 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.415 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48192/0 (socket says 172.21.15.156:48192) 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 -- 172.21.15.156:0/2355536912 learned_addr learned my addr 172.21.15.156:0/2355536912 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a4e43700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 0x7f04a0198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049ffff700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a0102470 0x7f04a019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 -- 172.21.15.156:0/2355536912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a0102470 msgr2=0x7f04a019e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a0102470 0x7f04a019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 -- 172.21.15.156:0/2355536912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 msgr2=0x7f04a0198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 0x7f04a0198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f049f7fe700 1 -- 172.21.15.156:0/2355536912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a0199060 con 0x7f04a0102e30 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.407+0000 7f04a4e43700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 0x7f04a0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f049f7fe700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f049400d950 tx=0x7f049400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f049d7fa700 1 -- 172.21.15.156:0/2355536912 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494016070 con 0x7f04a0102e30 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f049d7fa700 1 -- 172.21.15.156:0/2355536912 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0494010460 con 0x7f04a0102e30 2024-05-26T12:03:05.416 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a0199350 con 0x7f04a0102e30 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f049d7fa700 1 -- 172.21.15.156:0/2355536912 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049400f620 con 0x7f04a0102e30 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.408+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a01a5610 con 0x7f04a0102e30 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.410+0000 7f049d7fa700 1 -- 172.21.15.156:0/2355536912 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f0494019070 con 0x7f04a0102e30 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f049d7fa700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 0x7f04880b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f049ffff700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 0x7f04880b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f049d7fa700 1 -- 172.21.15.156:0/2355536912 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f04940e1f00 con 0x7f04a0102e30 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f049ffff700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 0x7f04880b4d40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04a0199a40 tx=0x7f0490005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f04b113cb80 1 Processor -- start 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.411+0000 7f04b113cb80 1 -- start start 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c41e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c429700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.417 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564d6c37bd20 con 0x564d6c429c40 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d6c36b8c0 con 0x564d6c41ed60 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04b113cb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d6c36f900 con 0x564d6c41e440 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48202/0 (socket says 172.21.15.156:48202) 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 -- 172.21.15.156:0/2619251233 learned_addr learned my addr 172.21.15.156:0/2619251233 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f049ffff700 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c41e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f049f7fe700 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c429700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 -- 172.21.15.156:0/2619251233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 msgr2=0x564d6c41e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c41e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 -- 172.21.15.156:0/2619251233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 msgr2=0x564d6c429700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c429700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.418 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 -- 172.21.15.156:0/2619251233 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d6c314f60 con 0x564d6c429c40 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04a4e43700 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0498006130 tx=0x7f04980090d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72058f01cdcb19c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04867fc700 1 -- 172.21.15.156:0/2619251233 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498020030 con 0x564d6c429c40 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04867fc700 1 -- 172.21.15.156:0/2619251233 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0498009370 con 0x564d6c429c40 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.412+0000 7f04867fc700 1 -- 172.21.15.156:0/2619251233 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049801c440 con 0x564d6c429c40 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 -- 172.21.15.156:0/2619251233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 msgr2=0x564d6c42c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f0498006130 tx=0x7f04980090d0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 -- 172.21.15.156:0/2619251233 shutdown_connections 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c42c030 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c41e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.419 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 --2- 172.21.15.156:0/2619251233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c429700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 -- 172.21.15.156:0/2619251233 >> 172.21.15.156:0/2619251233 conn(0x564d6c3491f0 msgr2=0x564d6c41adf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 -- 172.21.15.156:0/2619251233 shutdown_connections 2024-05-26T12:03:05.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.413+0000 7f04b113cb80 1 -- 172.21.15.156:0/2619251233 wait complete. 2024-05-26T12:03:05.420 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 -- start start 2024-05-26T12:03:05.423 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 Processor -- start 2024-05-26T12:03:05.423 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c429340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c426640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564d6c271f50 con 0x564d6c429c40 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d6c2d3d50 con 0x564d6c41ed60 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d6c42c910 con 0x564d6c41e440 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04b113cb80 10 client.0 _ll_register_callbacks cb 0x564d6c306900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48208/0 (socket says 172.21.15.156:48208) 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f04a4e43700 1 -- 172.21.15.156:0/616480532 learned_addr learned my addr 172.21.15.156:0/616480532 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f049f7fe700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c426640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.414+0000 7f049ffff700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c429340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 -- 172.21.15.156:0/616480532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 msgr2=0x564d6c429340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c429340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 -- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 msgr2=0x564d6c426640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c426640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 -- 172.21.15.156:0/616480532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d6c2bb880 con 0x564d6c429c40 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f049f7fe700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c426640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f049ffff700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c429340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04a4e43700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f049800d010 tx=0x7f0498007800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2dd5c3f1a343f95 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049802c070 con 0x564d6c429c40 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f0498004d10 con 0x564d6c429c40 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0498009370 con 0x564d6c429c40 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f0498038050 con 0x564d6c429c40 2024-05-26T12:03:05.425 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04b113cb80 10 client.15033 fetch_fsmap learned FSMap version 27 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564d6c325710 con 0x564d6c429c40 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0498009370 con 0x564d6c429c40 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.415+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f0498011070 con 0x564d6c429c40 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 10 client.15033 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 10 client.15033 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 20 client.15033 populate_metadata read hostname 'smithi156' 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c439010 0x564d6c43a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 4 client.15033 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 --> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564d6c37dc70 con 0x564d6c439010 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f049ffff700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c439010 0x564d6c43a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f049ffff700 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c439010 0x564d6c43a480 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6464a90087cfbf7 server_cookie=e81ec6b6589852d7 in_seq=0 out_seq=0 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.416+0000 7f047f7fe700 10 client.15033 ms_handle_connect on v2:172.21.15.186:6838/2121675127 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f047f7fe700 1 -- 172.21.15.156:0/616480532 <== mds.0 v2:172.21.15.186:6838/2121675127 1 ==== command_reply(tid 0: 0 ) ==== 8+0+24173 (crc 0 0 0) 0x564d6c37dc70 con 0x564d6c439010 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f047f7fe700 10 client.15033 handle_command_reply: tid=0 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 client.15033 shutdown 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 msgr2=0x564d6c426b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f049800d010 tx=0x7f0498007800 comp rx=0 tx=0).stop 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 shutdown_connections 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c439010 0x564d6c43a480 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c426b80 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c429340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f047f7fe700 0 client.15033 ms_handle_reset on v2:172.21.15.186:6838/2121675127 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 --2- 172.21.15.156:0/616480532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c426640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 >> 172.21.15.156:0/616480532 conn(0x564d6c3491f0 msgr2=0x564d6c417aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.417+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 shutdown_connections 2024-05-26T12:03:05.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.418+0000 7f04b113cb80 1 -- 172.21.15.156:0/616480532 wait complete. 2024-05-26T12:03:05.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.418+0000 7f04b113cb80 20 client.15033 trim_cache size 0 max 16384 2024-05-26T12:03:05.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 Processor -- start 2024-05-26T12:03:05.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 -- start start 2024-05-26T12:03:05.443 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c425bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c4216a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564d6c4387c0 con 0x564d6c429c40 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d6c438e90 con 0x564d6c41ed60 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.438+0000 7f04b113cb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d6c439560 con 0x564d6c41e440 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48212/0 (socket says 172.21.15.156:48212) 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 -- 172.21.15.156:0/3119804707 learned_addr learned my addr 172.21.15.156:0/3119804707 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f049ffff700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c425bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f049f7fe700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c4216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 -- 172.21.15.156:0/3119804707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 msgr2=0x564d6c425bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c425bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.444 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 -- 172.21.15.156:0/3119804707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 msgr2=0x564d6c4216a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c4216a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 -- 172.21.15.156:0/3119804707 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d6c438cb0 con 0x564d6c429c40 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f049f7fe700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c4216a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f049ffff700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c425bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f04a4e43700 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f049801c690 tx=0x7f049800b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c4d8bebdd16ca0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.439+0000 7f0485ffb700 1 -- 172.21.15.156:0/3119804707 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049802c070 con 0x564d6c429c40 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f0485ffb700 1 -- 172.21.15.156:0/3119804707 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049801cb70 con 0x564d6c429c40 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f0485ffb700 1 -- 172.21.15.156:0/3119804707 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0498020040 con 0x564d6c429c40 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 -- 172.21.15.156:0/3119804707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 msgr2=0x564d6c421be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f049801c690 tx=0x7f049800b440 comp rx=0 tx=0).stop 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 -- 172.21.15.156:0/3119804707 shutdown_connections 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c429c40 0x564d6c421be0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.445 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c41e440 0x564d6c425bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3119804707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41ed60 0x564d6c4216a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 -- 172.21.15.156:0/3119804707 >> 172.21.15.156:0/3119804707 conn(0x564d6c3491f0 msgr2=0x564d6c42c040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 -- 172.21.15.156:0/3119804707 shutdown_connections 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.440+0000 7f04b113cb80 1 -- 172.21.15.156:0/3119804707 wait complete. 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 -- start start 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 Processor -- start 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41e440 0x564d6c443900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c429c40 0x564d6c43c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.446 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x564d6c43a300 con 0x564d6c41ed60 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x564d6c439c30 con 0x564d6c41e440 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x564d6c43a9d0 con 0x564d6c429c40 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f04b113cb80 10 client.0 _ll_register_callbacks cb 0x564d6c306900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f049f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f049f7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48218/0 (socket says 172.21.15.156:48218) 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f049f7fe700 1 -- 172.21.15.156:0/3187373414 learned_addr learned my addr 172.21.15.156:0/3187373414 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.441+0000 7f049ffff700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41e440 0x564d6c443900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f04a4e43700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c429c40 0x564d6c43c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 -- 172.21.15.156:0/3187373414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c429c40 msgr2=0x564d6c43c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c429c40 0x564d6c43c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 -- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41e440 msgr2=0x564d6c443900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-26T12:03:05.447 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41e440 0x564d6c443900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 -- 172.21.15.156:0/3187373414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d6c43cde0 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f049f7fe700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f049400b2f0 tx=0x7f04940a9bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8ad36f3e5347f56 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f047f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.156:3300/0 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0494020040 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f049801cb70 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mon.0 v2:172.21.15.156:3300/0 2 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f04940a9750 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f04940b2350 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f04b113cb80 10 client.15039 fetch_fsmap learned FSMap version 27 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564d6c4a5050 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.442+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mon.0 v2:172.21.15.156:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f049801cb70 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mon.0 v2:172.21.15.156:3300/0 5 ==== fsmap(e 27) ==== 8354+0+0 (secure 0 0 0) 0x7f04940a8750 con 0x564d6c41ed60 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 10 client.15039 fetch_fsmap finished waiting for FSMap version 27 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 10 client.15039 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 20 client.15039 populate_metadata read hostname 'smithi156' 2024-05-26T12:03:05.448 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c4a9c60 0x564d6c4a8970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:05.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 4 client.15039 mds_command: new command op to 24476 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-26T12:03:05.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 --> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] -- command(tid 0: {"prefix": "status"}) -- 0x564d6c377450 con 0x564d6c4a9c60 2024-05-26T12:03:05.449 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f049ffff700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c4a9c60 0x564d6c4a8970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f047f7fe700 10 client.15039 ms_handle_connect on v2:172.21.15.186:6838/2121675127 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.443+0000 7f049ffff700 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c4a9c60 0x564d6c4a8970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e5a48fb597428c3 server_cookie=f33ef5d4080256ed in_seq=0 out_seq=0 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f047f7fe700 1 -- 172.21.15.156:0/3187373414 <== mds.0 v2:172.21.15.186:6838/2121675127 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564d6c377450 con 0x564d6c4a9c60 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f047f7fe700 10 client.15039 handle_command_reply: tid=0 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 client.15039 shutdown 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 msgr2=0x564d6c443e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f049400b2f0 tx=0x7f04940a9bf0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 shutdown_connections 2024-05-26T12:03:05.450 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] conn(0x564d6c4a9c60 0x564d6c4a8970 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x564d6c429c40 0x564d6c43c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564d6c41e440 0x564d6c443900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 --2- 172.21.15.156:0/3187373414 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x564d6c41ed60 0x564d6c443e40 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 >> 172.21.15.156:0/3187373414 conn(0x564d6c3491f0 msgr2=0x564d6c42c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f047f7fe700 0 client.15039 ms_handle_reset on v2:172.21.15.186:6838/2121675127 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 shutdown_connections 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 1 -- 172.21.15.156:0/3187373414 wait complete. 2024-05-26T12:03:05.451 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.444+0000 7f04b113cb80 20 client.15039 trim_cache size 0 max 16384 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout:{ 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "cluster_fsid": "38c1aff8-1b56-11ef-bc9b-c7b262605968", 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "whoami": 0, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "id": 24476, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "want_state": "up:active", 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "state": "up:active", 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "fs_name": "cephfs", 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "rank_uptime": 149.831596617, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "mdsmap_epoch": 27, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch": 83, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "osdmap_epoch_barrier": 79, 2024-05-26T12:03:05.452 INFO:teuthology.orchestra.run.smithi156.stdout: "uptime": 166.02588043200001 2024-05-26T12:03:05.453 INFO:teuthology.orchestra.run.smithi156.stdout:} 2024-05-26T12:03:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 msgr2=0x7f04880b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 0x7f04880b4d40 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f04a0199a40 tx=0x7f0490005ee0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.453 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 msgr2=0x7f04a019ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f049400d950 tx=0x7f049400dd10 comp rx=0 tx=0).stop 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 shutdown_connections 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f04880b2880 0x7f04880b4d40 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04a01084b0 0x7f04a0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f04a0102470 0x7f04a019e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 --2- 172.21.15.156:0/2355536912 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f04a0102e30 0x7f04a019ec80 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 >> 172.21.15.156:0/2355536912 conn(0x7f04a0079410 msgr2=0x7f04a01004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 shutdown_connections 2024-05-26T12:03:05.454 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:05.445+0000 7f04a68a6700 1 -- 172.21.15.156:0/2355536912 wait complete. 2024-05-26T12:03:06.236 INFO:tasks.cephadm:Setup complete, yielding 2024-05-26T12:03:06.237 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-26T12:03:06.247 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi156.front.sepia.ceph.com 2024-05-26T12:03:06.247 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch status' 2024-05-26T12:03:06.490 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:06.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:06 smithi156 ceph-mon[32025]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:06.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:06 smithi192 ceph-mon[36545]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:06.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:06 smithi186 ceph-mon[36412]: pgmap v372: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 -- 172.21.15.156:0/1236801580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f41036a0 msgr2=0x7f25f4103b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 --2- 172.21.15.156:0/1236801580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f41036a0 0x7f25f4103b00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f25e80099c0 tx=0x7f25e8009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 -- 172.21.15.156:0/1236801580 shutdown_connections 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 --2- 172.21.15.156:0/1236801580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f4104040 0x7f25f410a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 --2- 172.21.15.156:0/1236801580 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41024a0 0x7f25f41028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 --2- 172.21.15.156:0/1236801580 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f41036a0 0x7f25f4103b00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 -- 172.21.15.156:0/1236801580 >> 172.21.15.156:0/1236801580 conn(0x7f25f40fda60 msgr2=0x7f25f40ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 -- 172.21.15.156:0/1236801580 shutdown_connections 2024-05-26T12:03:08.291 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.277+0000 7f25fa70d700 1 -- 172.21.15.156:0/1236801580 wait complete. 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 Processor -- start 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 -- start start 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 0x7f25f41065d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41036a0 0x7f25f4104c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410dfc0 con 0x7f25f4104040 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410de40 con 0x7f25f41024a0 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25fa70d700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f25f410e140 con 0x7f25f41036a0 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25f8caa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.278+0000 7f25f8caa700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48236/0 (socket says 172.21.15.156:48236) 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 -- 172.21.15.156:0/2665389092 learned_addr learned my addr 172.21.15.156:0/2665389092 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f3fff700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 0x7f25f41065d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:08.292 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f37fe700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41036a0 0x7f25f4104c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 -- 172.21.15.156:0/2665389092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41036a0 msgr2=0x7f25f4104c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41036a0 0x7f25f4104c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 -- 172.21.15.156:0/2665389092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 msgr2=0x7f25f41065d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 0x7f25f41065d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 -- 172.21.15.156:0/2665389092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25f4105a10 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f3fff700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 0x7f25f41065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f8caa700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ada0 tx=0x7f25ec00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec017020 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25f41a3910 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f25ec00ce30 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25ec005350 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.279+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25f41a3e20 con 0x7f25f4104040 2024-05-26T12:03:08.293 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.280+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25f41a4050 con 0x7f25f4104040 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.282+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f25ec00c980 con 0x7f25f4104040 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.283+0000 7f25f17fa700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 0x7f25dc0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.283+0000 7f25f3fff700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 0x7f25dc0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.283+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f25ec0e1f70 con 0x7f25f4104040 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.283+0000 7f25f3fff700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 0x7f25dc0b4d40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25f4103500 tx=0x7f25e4008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:08.294 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.287+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f25ec021020 con 0x7f25f4104040 2024-05-26T12:03:08.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.507+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f25f41a4220 con 0x7f25dc0b2880 2024-05-26T12:03:08.508 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:08 smithi156 ceph-mon[32025]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:08.509 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.509+0000 7f25f17fa700 1 -- 172.21.15.156:0/2665389092 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f25f41a4220 con 0x7f25dc0b2880 2024-05-26T12:03:08.509 INFO:teuthology.orchestra.run.smithi156.stdout:Backend: cephadm 2024-05-26T12:03:08.509 INFO:teuthology.orchestra.run.smithi156.stdout:Available: Yes 2024-05-26T12:03:08.509 INFO:teuthology.orchestra.run.smithi156.stdout:Paused: No 2024-05-26T12:03:08.511 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 msgr2=0x7f25dc0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 0x7f25dc0b4d40 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f25f4103500 tx=0x7f25e4008040 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 msgr2=0x7f25f4105160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f25ec00ada0 tx=0x7f25ec00c330 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 shutdown_connections 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f25dc0b2880 0x7f25dc0b4d40 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f25f4104040 0x7f25f4105160 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25f41024a0 0x7f25f41065d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 --2- 172.21.15.156:0/2665389092 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f25f41036a0 0x7f25f4104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 >> 172.21.15.156:0/2665389092 conn(0x7f25f40fda60 msgr2=0x7f25f406bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 shutdown_connections 2024-05-26T12:03:08.512 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:08.511+0000 7f25fa70d700 1 -- 172.21.15.156:0/2665389092 wait complete. 2024-05-26T12:03:08.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:08 smithi192 ceph-mon[36545]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:08.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:08 smithi186 ceph-mon[36412]: pgmap v373: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:09.158 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-26T12:03:09.403 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:09.523 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:09 smithi156 ceph-mon[32025]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:09.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:09 smithi192 ceph-mon[36545]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:09.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:09 smithi186 ceph-mon[36412]: from='client.15045 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:10.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:10 smithi156 ceph-mon[32025]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:10.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:10 smithi192 ceph-mon[36545]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:10.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:10 smithi186 ceph-mon[36412]: pgmap v374: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:11.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 -- 172.21.15.156:0/2736023069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1047a0 msgr2=0x7f155c104c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:11.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 --2- 172.21.15.156:0/2736023069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1047a0 0x7f155c104c00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f154c0099c0 tx=0x7f154c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 -- 172.21.15.156:0/2736023069 shutdown_connections 2024-05-26T12:03:11.185 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 --2- 172.21.15.156:0/2736023069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c105140 0x7f155c10b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 --2- 172.21.15.156:0/2736023069 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c1035a0 0x7f155c1039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 --2- 172.21.15.156:0/2736023069 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1047a0 0x7f155c104c00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 -- 172.21.15.156:0/2736023069 >> 172.21.15.156:0/2736023069 conn(0x7f155c0feb40 msgr2=0x7f155c100f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 -- 172.21.15.156:0/2736023069 shutdown_connections 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.183+0000 7f1563829700 1 -- 172.21.15.156:0/2736023069 wait complete. 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 Processor -- start 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 -- start start 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c1047a0 0x7f155c075930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 0x7f155c075e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10f130 con 0x7f155c1035a0 2024-05-26T12:03:11.186 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10efb0 con 0x7f155c1047a0 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1563829700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f155c10f2b0 con 0x7f155c105140 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48254/0 (socket says 172.21.15.156:48254) 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 -- 172.21.15.156:0/1111375966 learned_addr learned my addr 172.21.15.156:0/1111375966 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1560dc4700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c1047a0 0x7f155c075930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1561dc6700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 0x7f155c075e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 -- 172.21.15.156:0/1111375966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 msgr2=0x7f155c075e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 0x7f155c075e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 -- 172.21.15.156:0/1111375966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c1047a0 msgr2=0x7f155c075930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c1047a0 0x7f155c075930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 -- 172.21.15.156:0/1111375966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f155c076720 con 0x7f155c1035a0 2024-05-26T12:03:11.187 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f1561dc6700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 0x7f155c075e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15615c5700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f155800b500 tx=0x7f155800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.184+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558015020 con 0x7f155c1035a0 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.185+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f155c1a3950 con 0x7f155c1035a0 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.185+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1558011ce0 con 0x7f155c1035a0 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.185+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1558010d60 con 0x7f155c1035a0 2024-05-26T12:03:11.188 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.185+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f155c1a3f10 con 0x7f155c1035a0 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.186+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f155c06a230 con 0x7f155c1035a0 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.187+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f1558018020 con 0x7f155c1035a0 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.187+0000 7f15527fc700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 0x7f15480b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.188+0000 7f1560dc4700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 0x7f15480b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.188+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f15580e1fd0 con 0x7f155c1035a0 2024-05-26T12:03:11.189 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.188+0000 7f1560dc4700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 0x7f15480b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f154c000c00 tx=0x7f154c005fb0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:11.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.191+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f155801f020 con 0x7f155c1035a0 2024-05-26T12:03:11.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.410+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f155c063980 con 0x7f15480b2880 2024-05-26T12:03:11.424 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.424+0000 7f15527fc700 1 -- 172.21.15.156:0/1111375966 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f155c063980 con 0x7f15480b2880 2024-05-26T12:03:11.424 INFO:teuthology.orchestra.run.smithi156.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-26T12:03:11.424 INFO:teuthology.orchestra.run.smithi156.stdout:mds.a smithi156 running (2m) 34s ago 2m 19.2M - 19.0.0-3867-g9a4c5853 939e725847fb e0f4feea93d8 2024-05-26T12:03:11.424 INFO:teuthology.orchestra.run.smithi156.stdout:mds.b smithi186 running (2m) 36s ago 2m 26.0M - 19.0.0-3867-g9a4c5853 939e725847fb 30a3efad7f4e 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.c smithi192 running (3m) 36s ago 3m 20.1M - 19.0.0-3867-g9a4c5853 939e725847fb b43d74acfb02 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.d smithi156 running (3m) 34s ago 3m 14.4M - 19.0.0-3867-g9a4c5853 939e725847fb 0f0a5733d02a 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.e smithi186 running (3m) 36s ago 3m 19.8M - 19.0.0-3867-g9a4c5853 939e725847fb 71176291a1a0 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.f smithi192 running (3m) 36s ago 3m 21.5M - 19.0.0-3867-g9a4c5853 939e725847fb 2e245d9e4126 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.g smithi156 running (3m) 34s ago 2m 20.7M - 19.0.0-3867-g9a4c5853 939e725847fb 8d8891dfe154 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.h smithi186 running (3m) 36s ago 3m 22.0M - 19.0.0-3867-g9a4c5853 939e725847fb b2fbf4ed1723 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.i smithi192 running (3m) 36s ago 3m 21.8M - 19.0.0-3867-g9a4c5853 939e725847fb 0fca86089d87 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.j smithi156 running (3m) 34s ago 3m 14.5M - 19.0.0-3867-g9a4c5853 939e725847fb 134f94b5d224 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.k smithi186 running (3m) 36s ago 3m 22.2M - 19.0.0-3867-g9a4c5853 939e725847fb 84620862ab39 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mds.l smithi192 running (3m) 36s ago 3m 21.7M - 19.0.0-3867-g9a4c5853 939e725847fb ba3e175c640b 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mgr.x smithi156 *:9283,8765 running (10m) 34s ago 10m 530M - 19.0.0-3867-g9a4c5853 939e725847fb 9bb67b950b96 2024-05-26T12:03:11.425 INFO:teuthology.orchestra.run.smithi156.stdout:mgr.y smithi186 *:8443,8765 running (8m) 36s ago 8m 468M - 19.0.0-3867-g9a4c5853 939e725847fb 05b15e8d722f 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:mgr.z smithi192 *:8443,8765 running (8m) 36s ago 8m 467M - 19.0.0-3867-g9a4c5853 939e725847fb 966d610e70fc 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:mon.a smithi156 running (10m) 34s ago 11m 55.6M 2048M 19.0.0-3867-g9a4c5853 939e725847fb 3cc8157ae7b0 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:mon.b smithi186 running (9m) 36s ago 9m 39.6M 2048M 19.0.0-3867-g9a4c5853 939e725847fb 3991ed6389a2 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:mon.c smithi192 running (9m) 36s ago 9m 43.0M 2048M 19.0.0-3867-g9a4c5853 939e725847fb a527f7e70c41 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.0 smithi156 running (8m) 34s ago 8m 56.3M 4272M 19.0.0-3867-g9a4c5853 939e725847fb d98985ed8b13 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.1 smithi186 running (8m) 36s ago 8m 45.3M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 61df0fb6d7c7 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.2 smithi192 running (7m) 36s ago 7m 48.9M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 5e82e4b68fd1 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.3 smithi156 running (7m) 34s ago 7m 43.7M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 43f94096b45b 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.4 smithi186 running (6m) 36s ago 6m 53.7M 4272M 19.0.0-3867-g9a4c5853 939e725847fb de1c2d81fede 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.5 smithi192 running (6m) 36s ago 6m 54.7M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 5fa9edc10d39 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.6 smithi156 running (6m) 34s ago 6m 44.0M 4272M 19.0.0-3867-g9a4c5853 939e725847fb f9967b0e01f5 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.7 smithi186 running (5m) 36s ago 5m 43.3M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 0eb8760bbfee 2024-05-26T12:03:11.426 INFO:teuthology.orchestra.run.smithi156.stdout:osd.8 smithi192 running (5m) 36s ago 5m 43.8M 4272M 19.0.0-3867-g9a4c5853 939e725847fb ca57e492b83f 2024-05-26T12:03:11.427 INFO:teuthology.orchestra.run.smithi156.stdout:osd.9 smithi156 running (4m) 34s ago 4m 43.5M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 26127fd2f910 2024-05-26T12:03:11.427 INFO:teuthology.orchestra.run.smithi156.stdout:osd.10 smithi186 running (4m) 36s ago 4m 44.3M 4272M 19.0.0-3867-g9a4c5853 939e725847fb 2b96e9b52270 2024-05-26T12:03:11.427 INFO:teuthology.orchestra.run.smithi156.stdout:osd.11 smithi192 running (3m) 36s ago 3m 53.7M 4272M 19.0.0-3867-g9a4c5853 939e725847fb a54259b3109f 2024-05-26T12:03:11.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 msgr2=0x7f15480b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 0x7f15480b4d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f154c000c00 tx=0x7f154c005fb0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 msgr2=0x7f155c0772e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f155800b500 tx=0x7f155800b8c0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 shutdown_connections 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f15480b2880 0x7f15480b4d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f155c105140 0x7f155c075e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f155c1035a0 0x7f155c0772e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 --2- 172.21.15.156:0/1111375966 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f155c1047a0 0x7f155c075930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 >> 172.21.15.156:0/1111375966 conn(0x7f155c0feb40 msgr2=0x7f155c107b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 shutdown_connections 2024-05-26T12:03:11.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:11.426+0000 7f1563829700 1 -- 172.21.15.156:0/1111375966 wait complete. 2024-05-26T12:03:12.240 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-26T12:03:12.485 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:12.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:12 smithi156 ceph-mon[32025]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:12.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:12 smithi156 ceph-mon[32025]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:12 smithi186 ceph-mon[36412]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:12.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:12 smithi186 ceph-mon[36412]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:12.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:12 smithi192 ceph-mon[36545]: pgmap v375: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:12.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:12 smithi192 ceph-mon[36545]: from='client.15051 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 -- 172.21.15.156:0/236055766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c1024a0 msgr2=0x7fc76c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:14.128 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 --2- 172.21.15.156:0/236055766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c1024a0 0x7fc76c1028c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fc75c0099c0 tx=0x7fc75c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 -- 172.21.15.156:0/236055766 shutdown_connections 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 --2- 172.21.15.156:0/236055766 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c104040 0x7fc76c10a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 --2- 172.21.15.156:0/236055766 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c1024a0 0x7fc76c1028c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 --2- 172.21.15.156:0/236055766 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1036a0 0x7fc76c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 -- 172.21.15.156:0/236055766 >> 172.21.15.156:0/236055766 conn(0x7fc76c0fda40 msgr2=0x7fc76c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 -- 172.21.15.156:0/236055766 shutdown_connections 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 -- 172.21.15.156:0/236055766 wait complete. 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.126+0000 7fc77248b700 1 Processor -- start 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 -- start start 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1024a0 0x7fc76c1984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 0x7fc76c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:14.129 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c10df30 con 0x7fc76c104040 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c10ddb0 con 0x7fc76c1024a0 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc77248b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c10e0b0 con 0x7fc76c1036a0 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:48274/0 (socket says 172.21.15.156:48274) 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 -- 172.21.15.156:0/2843219973 learned_addr learned my addr 172.21.15.156:0/2843219973 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc76bfff700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1024a0 0x7fc76c1984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc763fff700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 0x7fc76c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 -- 172.21.15.156:0/2843219973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 msgr2=0x7fc76c198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 0x7fc76c198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 -- 172.21.15.156:0/2843219973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1024a0 msgr2=0x7fc76c1984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:14.130 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1024a0 0x7fc76c1984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 -- 172.21.15.156:0/2843219973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c1a0150 con 0x7fc76c104040 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc763fff700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 0x7fc76c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc770a28700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc764007da0 tx=0x7fc76400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.127+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc764016020 con 0x7fc76c104040 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.128+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc76400ad00 con 0x7fc76c104040 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.128+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c1a0440 con 0x7fc76c104040 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.128+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7640115e0 con 0x7fc76c104040 2024-05-26T12:03:14.131 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.128+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c1a0930 con 0x7fc76c104040 2024-05-26T12:03:14.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.129+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc76c06a230 con 0x7fc76c104040 2024-05-26T12:03:14.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.130+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fc764019020 con 0x7fc76c104040 2024-05-26T12:03:14.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.131+0000 7fc769ffb700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 0x7fc7580b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:14.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.131+0000 7fc76bfff700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 0x7fc7580b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:14.132 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.131+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fc7640e2290 con 0x7fc76c104040 2024-05-26T12:03:14.133 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.131+0000 7fc76bfff700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 0x7fc7580b4d40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc75c003790 tx=0x7fc75c003860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:14.136 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.136+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fc764020020 con 0x7fc76c104040 2024-05-26T12:03:14.355 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.355+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fc76c063980 con 0x7fc7580b2880 2024-05-26T12:03:14.361 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.361+0000 7fc769ffb700 1 -- 172.21.15.156:0/2843219973 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fc76c063980 con 0x7fc7580b2880 2024-05-26T12:03:14.361 INFO:teuthology.orchestra.run.smithi156.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-26T12:03:14.362 INFO:teuthology.orchestra.run.smithi156.stdout:mds.all 12/12 39s ago 3m smithi156=a;smithi156=d;smithi156=g;smithi156=j;smithi186=b;smithi186=e;smithi186=h;smithi186=k;smithi192=c;smithi192=f;smithi192=i;smithi192=l;count:12 2024-05-26T12:03:14.362 INFO:teuthology.orchestra.run.smithi156.stdout:mgr 3/3 39s ago 9m smithi156=x;smithi186=y;smithi192=z;count:3 2024-05-26T12:03:14.362 INFO:teuthology.orchestra.run.smithi156.stdout:mon 3/3 39s ago 9m smithi156:172.21.15.156=a;smithi186:172.21.15.186=b;smithi192:172.21.15.192=c;count:3 2024-05-26T12:03:14.362 INFO:teuthology.orchestra.run.smithi156.stdout:osd 12 39s ago - 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 msgr2=0x7fc7580b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 0x7fc7580b4d40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc75c003790 tx=0x7fc75c003860 comp rx=0 tx=0).stop 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 msgr2=0x7fc76c19fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fc764007da0 tx=0x7fc76400a340 comp rx=0 tx=0).stop 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 shutdown_connections 2024-05-26T12:03:14.364 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fc7580b2880 0x7fc7580b4d40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fc76c104040 0x7fc76c19fab0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc76c1024a0 0x7fc76c1984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 --2- 172.21.15.156:0/2843219973 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fc76c1036a0 0x7fc76c198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 >> 172.21.15.156:0/2843219973 conn(0x7fc76c0fda40 msgr2=0x7fc76c1069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 shutdown_connections 2024-05-26T12:03:14.365 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:14.363+0000 7fc77248b700 1 -- 172.21.15.156:0/2843219973 wait complete. 2024-05-26T12:03:14.694 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:14 smithi156 ceph-mon[32025]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:14.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:14 smithi186 ceph-mon[36412]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:14.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:14 smithi192 ceph-mon[36545]: pgmap v376: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:15.029 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-26T12:03:15.273 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:15.657 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:15 smithi156 ceph-mon[32025]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:15.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:15 smithi186 ceph-mon[36412]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:15.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:15 smithi192 ceph-mon[36545]: from='client.15057 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:16.438 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:16 smithi156 ceph-mon[32025]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:16.627 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:16 smithi186 ceph-mon[36412]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:16.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:16 smithi192 ceph-mon[36545]: pgmap v377: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 -- 172.21.15.156:0/696031383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c1024a0 msgr2=0x7fb45c1028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 --2- 172.21.15.156:0/696031383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c1024a0 0x7fb45c1028c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fb44c0099c0 tx=0x7fb44c009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 -- 172.21.15.156:0/696031383 shutdown_connections 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 --2- 172.21.15.156:0/696031383 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c104040 0x7fb45c10a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 --2- 172.21.15.156:0/696031383 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c1024a0 0x7fb45c1028c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 --2- 172.21.15.156:0/696031383 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 0x7fb45c103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 -- 172.21.15.156:0/696031383 >> 172.21.15.156:0/696031383 conn(0x7fb45c0fda40 msgr2=0x7fb45c0ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 -- 172.21.15.156:0/696031383 shutdown_connections 2024-05-26T12:03:17.001 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.971+0000 7fb464195700 1 -- 172.21.15.156:0/696031383 wait complete. 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 Processor -- start 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 -- start start 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c1024a0 0x7fb45c06bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 0x7fb45c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10dfe0 con 0x7fb45c104040 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10de60 con 0x7fb45c1024a0 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb464195700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c10e160 con 0x7fb45c1036a0 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb462732700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb462732700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43476/0 (socket says 172.21.15.156:43476) 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb462732700 1 -- 172.21.15.156:0/1929211111 learned_addr learned my addr 172.21.15.156:0/1929211111 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:17.002 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.972+0000 7fb461f31700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c1024a0 0x7fb45c06bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb461730700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 0x7fb45c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 -- 172.21.15.156:0/1929211111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 msgr2=0x7fb45c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 0x7fb45c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 -- 172.21.15.156:0/1929211111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c1024a0 msgr2=0x7fb45c06bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c1024a0 0x7fb45c06bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 -- 172.21.15.156:0/1929211111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c1a1470 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb462732700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb45400ada0 tx=0x7fb45400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454017020 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c1a1760 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb45400ce30 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb454005350 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.973+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1a1ca0 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.974+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c06a230 con 0x7fb45c104040 2024-05-26T12:03:17.003 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.975+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fb45400c980 con 0x7fb45c104040 2024-05-26T12:03:17.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.976+0000 7fb452ffd700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 0x7fb4480b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:17.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.976+0000 7fb461f31700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 0x7fb4480b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:17.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.976+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fb4540e1f10 con 0x7fb45c104040 2024-05-26T12:03:17.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.976+0000 7fb461f31700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 0x7fb4480b4d40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006010 tx=0x7fb44c003860 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:17.004 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:16.979+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fb454021020 con 0x7fb45c104040 2024-05-26T12:03:17.191 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.190+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fb45c063980 con 0x7fb4480b2880 2024-05-26T12:03:17.192 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.192+0000 7fb452ffd700 1 -- 172.21.15.156:0/1929211111 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fb45c063980 con 0x7fb4480b2880 2024-05-26T12:03:17.192 INFO:teuthology.orchestra.run.smithi156.stdout:HOST ADDR LABELS STATUS 2024-05-26T12:03:17.193 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156 172.21.15.156 2024-05-26T12:03:17.193 INFO:teuthology.orchestra.run.smithi156.stdout:smithi186 172.21.15.186 2024-05-26T12:03:17.193 INFO:teuthology.orchestra.run.smithi156.stdout:smithi192 172.21.15.192 2024-05-26T12:03:17.193 INFO:teuthology.orchestra.run.smithi156.stdout:3 hosts in cluster 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 msgr2=0x7fb4480b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 0x7fb4480b4d40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb44c006010 tx=0x7fb44c003860 comp rx=0 tx=0).stop 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 msgr2=0x7fb45c1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fb45400ada0 tx=0x7fb45400c330 comp rx=0 tx=0).stop 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 shutdown_connections 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fb4480b2880 0x7fb4480b4d40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fb45c104040 0x7fb45c1a0ba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.195 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb45c1024a0 0x7fb45c06bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 --2- 172.21.15.156:0/1929211111 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fb45c1036a0 0x7fb45c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:17.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 >> 172.21.15.156:0/1929211111 conn(0x7fb45c0fda40 msgr2=0x7fb45c106ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:17.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 shutdown_connections 2024-05-26T12:03:17.196 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:17.194+0000 7fb464195700 1 -- 172.21.15.156:0/1929211111 wait complete. 2024-05-26T12:03:17.972 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-26T12:03:18.218 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:18.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:18 smithi156 ceph-mon[32025]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:18.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:18 smithi156 ceph-mon[32025]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:18 smithi186 ceph-mon[36412]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:18.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:18 smithi186 ceph-mon[36412]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:18 smithi192 ceph-mon[36545]: pgmap v378: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:18.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:18 smithi192 ceph-mon[36545]: from='client.15063 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- 172.21.15.156:0/2199111840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103100 msgr2=0x7ff44810a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 --2- 172.21.15.156:0/2199111840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103100 0x7ff44810a9a0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff44000b0a0 tx=0x7ff44000b3b0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- 172.21.15.156:0/2199111840 shutdown_connections 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 --2- 172.21.15.156:0/2199111840 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103100 0x7ff44810a9a0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 --2- 172.21.15.156:0/2199111840 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448103ae0 0x7ff448103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 --2- 172.21.15.156:0/2199111840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448102740 0x7ff448102bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- 172.21.15.156:0/2199111840 >> 172.21.15.156:0/2199111840 conn(0x7ff4480fd1f0 msgr2=0x7ff4480ff630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- 172.21.15.156:0/2199111840 shutdown_connections 2024-05-26T12:03:19.993 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- 172.21.15.156:0/2199111840 wait complete. 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 Processor -- start 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.977+0000 7ff45008b700 1 -- start start 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 0x7ff448102190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448103100 0x7ff4481a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7ff44810dfd0 con 0x7ff448103ae0 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7ff44810de50 con 0x7ff448103100 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7ff44810e150 con 0x7ff448102740 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43490/0 (socket says 172.21.15.156:43490) 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 -- 172.21.15.156:0/2446620387 learned_addr learned my addr 172.21.15.156:0/2446620387 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44d626700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448103100 0x7ff4481a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:19.994 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44de27700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 0x7ff448102190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 -- 172.21.15.156:0/2446620387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 msgr2=0x7ff448102190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 0x7ff448102190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 -- 172.21.15.156:0/2446620387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448103100 msgr2=0x7ff4481a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448103100 0x7ff4481a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 -- 172.21.15.156:0/2446620387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4481a34b0 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44de27700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 0x7ff448102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff44e628700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff44000ced0 tx=0x7ff44000cfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440021070 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff44001d900 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4481a3740 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff440005420 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.978+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4481a3d50 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.979+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff44806a230 con 0x7ff448103ae0 2024-05-26T12:03:19.995 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.981+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7ff440024070 con 0x7ff448103ae0 2024-05-26T12:03:19.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.982+0000 7ff43effd700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 0x7ff4340b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:19.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.982+0000 7ff44de27700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 0x7ff4340b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:19.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.982+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7ff44000f020 con 0x7ff448103ae0 2024-05-26T12:03:19.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.982+0000 7ff44de27700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 0x7ff4340b4dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff44806ba90 tx=0x7ff444008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:19.996 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:19.985+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7ff44002b020 con 0x7ff448103ae0 2024-05-26T12:03:20.198 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.197+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 --> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7ff448063980 con 0x7ff4340b2900 2024-05-26T12:03:20.199 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.199+0000 7ff43effd700 1 -- 172.21.15.156:0/2446620387 <== mgr.14150 v2:172.21.15.156:6800/3256012970 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7ff448063980 con 0x7ff4340b2900 2024-05-26T12:03:20.200 INFO:teuthology.orchestra.run.smithi156.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-26T12:03:20.200 INFO:teuthology.orchestra.run.smithi156.stdout:smithi156 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB74760040480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-05-26T12:03:20.200 INFO:teuthology.orchestra.run.smithi156.stdout:smithi186 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300CB400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-05-26T12:03:20.200 INFO:teuthology.orchestra.run.smithi156.stdout:smithi192 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500NN400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-26T12:03:20.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 msgr2=0x7ff4340b4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:20.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 0x7ff4340b4dc0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7ff44806ba90 tx=0x7ff444008040 comp rx=0 tx=0).stop 2024-05-26T12:03:20.202 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 msgr2=0x7ff4481a2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7ff44000ced0 tx=0x7ff44000cfb0 comp rx=0 tx=0).stop 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 shutdown_connections 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7ff4340b2900 0x7ff4340b4dc0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7ff448103ae0 0x7ff4481a2cd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7ff448102740 0x7ff448102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 --2- 172.21.15.156:0/2446620387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff448103100 0x7ff4481a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 >> 172.21.15.156:0/2446620387 conn(0x7ff4480fd1f0 msgr2=0x7ff448106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 shutdown_connections 2024-05-26T12:03:20.203 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:20.202+0000 7ff45008b700 1 -- 172.21.15.156:0/2446620387 wait complete. 2024-05-26T12:03:20.623 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:20 smithi192 ceph-mon[36545]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:20.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:20 smithi186 ceph-mon[36412]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:20.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:20 smithi156 ceph-mon[32025]: pgmap v379: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:20.999 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-26T12:03:21.009 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi156.front.sepia.ceph.com 2024-05-26T12:03:21.009 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-26T12:03:21.253 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:21.624 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:21 smithi156 ceph-mon[32025]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:21.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:21 smithi186 ceph-mon[36412]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:21.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:21 smithi192 ceph-mon[36545]: from='client.15069 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-26T12:03:22.697 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:22 smithi156 ceph-mon[32025]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:22.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:22 smithi186 ceph-mon[36412]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:22.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:22 smithi192 ceph-mon[36545]: pgmap v380: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:23.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.058+0000 7fdf95098700 1 -- 172.21.15.156:0/4000351022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf90103a80 msgr2=0x7fdf90103ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:23.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.058+0000 7fdf95098700 1 --2- 172.21.15.156:0/4000351022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf90103a80 0x7fdf90103ea0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fdf800099c0 tx=0x7fdf80009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 -- 172.21.15.156:0/4000351022 shutdown_connections 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 --2- 172.21.15.156:0/4000351022 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103160 0x7fdf9010aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 --2- 172.21.15.156:0/4000351022 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf90103a80 0x7fdf90103ea0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 --2- 172.21.15.156:0/4000351022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf901027c0 0x7fdf90102c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 -- 172.21.15.156:0/4000351022 >> 172.21.15.156:0/4000351022 conn(0x7fdf900fd210 msgr2=0x7fdf900ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 -- 172.21.15.156:0/4000351022 shutdown_connections 2024-05-26T12:03:23.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 -- 172.21.15.156:0/4000351022 wait complete. 2024-05-26T12:03:23.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.059+0000 7fdf95098700 1 Processor -- start 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 -- start start 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf90103160 0x7fdf9019e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103a80 0x7fdf901a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010df90 con 0x7fdf901027c0 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010de10 con 0x7fdf90103160 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf95098700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fdf9010e110 con 0x7fdf90103a80 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:43516/0 (socket says 172.21.15.156:43516) 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 -- 172.21.15.156:0/1154187896 learned_addr learned my addr 172.21.15.156:0/1154187896 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:23.062 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8f59e700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103a80 0x7fdf901a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8e59c700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf90103160 0x7fdf9019e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 -- 172.21.15.156:0/1154187896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103a80 msgr2=0x7fdf901a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103a80 0x7fdf901a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 -- 172.21.15.156:0/1154187896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf90103160 msgr2=0x7fdf9019e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf90103160 0x7fdf9019e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 -- 172.21.15.156:0/1154187896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf901a1360 con 0x7fdf901027c0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf8ed9d700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf800038c0 tx=0x7fdf800047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.060+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80027070 con 0x7fdf901027c0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.061+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf901a15f0 con 0x7fdf901027c0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.061+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdf80004160 con 0x7fdf901027c0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.061+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf80022e20 con 0x7fdf901027c0 2024-05-26T12:03:23.063 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.061+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf901a1b20 con 0x7fdf901027c0 2024-05-26T12:03:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.062+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf9006a230 con 0x7fdf901027c0 2024-05-26T12:03:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.063+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fdf8002a070 con 0x7fdf901027c0 2024-05-26T12:03:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.064+0000 7fdf77fff700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 0x7fdf780b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.064+0000 7fdf8e59c700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 0x7fdf780b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:23.064 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.064+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7fdf800e2e70 con 0x7fdf901027c0 2024-05-26T12:03:23.065 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.064+0000 7fdf8e59c700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 0x7fdf780b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005950 tx=0x7fdf840058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:23.068 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.067+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fdf80013070 con 0x7fdf901027c0 2024-05-26T12:03:23.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.315+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fdf9002e560 con 0x7fdf901027c0 2024-05-26T12:03:23.316 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.315+0000 7fdf77fff700 1 -- 172.21.15.156:0/1154187896 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3104 (secure 0 0 0) 0x7fdf800aabf0 con 0x7fdf901027c0 2024-05-26T12:03:23.316 INFO:teuthology.orchestra.run.smithi156.stdout:e27 2024-05-26T12:03:23.316 INFO:teuthology.orchestra.run.smithi156.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-26T12:03:23.316 INFO:teuthology.orchestra.run.smithi156.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:legacy client fscid: 1 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:Filesystem 'cephfs' (1) 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:fs_name cephfs 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 27 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:created 2024-05-26T12:00:34.689787+0000 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:modified 2024-05-26T12:00:55.501071+0000 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:tableserver 0 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:root 0 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:session_timeout 300 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:session_autoclose 300 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:max_file_size 1099511627776 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:max_xattr_size 65536 2024-05-26T12:03:23.317 INFO:teuthology.orchestra.run.smithi156.stdout:required_client_features {} 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:last_failure 0 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:last_failure_osd_epoch 0 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:max_mds 5 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:in 0,1,2,3,4 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:up {0=24476,1=24458,2=24449,3=24442,4=24431} 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:failed 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:damaged 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:stopped 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:data_pools [3] 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:metadata_pool 2 2024-05-26T12:03:23.318 INFO:teuthology.orchestra.run.smithi156.stdout:inline_data disabled 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:balancer 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:bal_rank_mask -1 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:standby_count_wanted 0 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:qdb_cluster leader: 24476 members: 24458,24476,24431,24442,24449 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.b{0:24476} state up:active seq 6 addr [v2:172.21.15.186:6838/2121675127,v1:172.21.15.186:6839/2121675127] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.c{0:24427} state up:standby-replay seq 1 addr [v2:172.21.15.192:6836/977346748,v1:172.21.15.192:6837/977346748] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.h{1:24458} state up:active seq 13 addr [v2:172.21.15.186:6836/3519203877,v1:172.21.15.186:6837/3519203877] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.f{1:24418} state up:standby-replay seq 1 addr [v2:172.21.15.192:6834/699682135,v1:172.21.15.192:6835/699682135] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.e{2:24449} state up:active seq 15 addr [v2:172.21.15.186:6834/1012390034,v1:172.21.15.186:6835/1012390034] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.l{2:24397} state up:standby-replay seq 1 addr [v2:172.21.15.192:6832/4217003952,v1:172.21.15.192:6833/4217003952] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.i{3:24442} state up:active seq 12 addr [v2:172.21.15.192:6838/1614885810,v1:172.21.15.192:6839/1614885810] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.a{3:14613} state up:standby-replay seq 1 addr [v2:172.21.15.156:6840/1148384159,v1:172.21.15.156:6841/1148384159] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.319 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.k{4:24431} state up:active seq 18 addr [v2:172.21.15.186:6832/586672120,v1:172.21.15.186:6833/586672120] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.g{4:14601} state up:standby-replay seq 1 addr [v2:172.21.15.156:6838/2826647780,v1:172.21.15.156:6839/2826647780] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout:Standby daemons: 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.156:6834/1321300121,v1:172.21.15.156:6835/1321300121] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.320 INFO:teuthology.orchestra.run.smithi156.stdout:[mds.d{-1:14547} state up:standby seq 1 addr [v2:172.21.15.156:6836/2020460154,v1:172.21.15.156:6837/2020460154] compat {c=[1],r=[1],i=[1fff]}] 2024-05-26T12:03:23.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 msgr2=0x7fdf780b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:23.321 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 0x7fdf780b4d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fdf84005950 tx=0x7fdf840058e0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 msgr2=0x7fdf90102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fdf800038c0 tx=0x7fdf800047b0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 shutdown_connections 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fdf780b2880 0x7fdf780b4d40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fdf90103a80 0x7fdf901a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fdf901027c0 0x7fdf90102230 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 --2- 172.21.15.156:0/1154187896 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf90103160 0x7fdf9019e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 >> 172.21.15.156:0/1154187896 conn(0x7fdf900fd210 msgr2=0x7fdf90106ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 shutdown_connections 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:23.318+0000 7fdf95098700 1 -- 172.21.15.156:0/1154187896 wait complete. 2024-05-26T12:03:23.322 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:03:23.685 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:23 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1154187896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-26T12:03:23.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:23 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1154187896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-26T12:03:23.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:23 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1154187896' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-26T12:03:24.132 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-26T12:03:24.375 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:24.497 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:24 smithi156 ceph-mon[32025]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:24.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:24 smithi186 ceph-mon[36412]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:24.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:24 smithi192 ceph-mon[36545]: pgmap v381: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:26.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 -- 172.21.15.156:0/588725781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 msgr2=0x7f111410a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:26.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 --2- 172.21.15.156:0/588725781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f111410a8f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b0d0 tx=0x7f110c00b3e0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.045 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 -- 172.21.15.156:0/588725781 shutdown_connections 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 --2- 172.21.15.156:0/588725781 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f111410a8f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 --2- 172.21.15.156:0/588725781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141024a0 0x7f11141028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 --2- 172.21.15.156:0/588725781 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141036a0 0x7f1114103b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.042+0000 7f111bca3700 1 -- 172.21.15.156:0/588725781 >> 172.21.15.156:0/588725781 conn(0x7f11140fda40 msgr2=0x7f11140ffe80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- 172.21.15.156:0/588725781 shutdown_connections 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- 172.21.15.156:0/588725781 wait complete. 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 Processor -- start 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- start start 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 0x7f111406bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141036a0 0x7f111419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:26.046 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f111410df20 con 0x7f1114104040 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f111410dda0 con 0x7f11141036a0 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.043+0000 7f111bca3700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f111410e0a0 con 0x7f11141024a0 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36668/0 (socket says 172.21.15.156:36668) 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 -- 172.21.15.156:0/1318895090 learned_addr learned my addr 172.21.15.156:0/1318895090 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111923e700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141036a0 0x7f111419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f1119a3f700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 0x7f111406bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 -- 172.21.15.156:0/1318895090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 msgr2=0x7f111406bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 0x7f111406bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 -- 172.21.15.156:0/1318895090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141036a0 msgr2=0x7f111419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141036a0 0x7f111419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 -- 172.21.15.156:0/1318895090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11141a1320 con 0x7f1114104040 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f1119a3f700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 0x7f111406bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:26.047 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111a240700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d010 tx=0x7f110c007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:26.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c021070 con 0x7f1114104040 2024-05-26T12:03:26.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11141a15b0 con 0x7f1114104040 2024-05-26T12:03:26.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c01db00 con 0x7f1114104040 2024-05-26T12:03:26.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c01bb60 con 0x7f1114104040 2024-05-26T12:03:26.048 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.044+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11141a1ab0 con 0x7f1114104040 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.045+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f111406a230 con 0x7f1114104040 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.046+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f110c024070 con 0x7f1114104040 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.047+0000 7f110affd700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 0x7f11000b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.047+0000 7f1119a3f700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 0x7f11000b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.047+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f110c00f020 con 0x7f1114104040 2024-05-26T12:03:26.049 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.048+0000 7f1119a3f700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 0x7f11000b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1110005950 tx=0x7f11100058e0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:26.052 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.052+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f110c02b020 con 0x7f1114104040 2024-05-26T12:03:26.256 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.256+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f1114064150 con 0x7f1114104040 2024-05-26T12:03:26.257 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.257+0000 7f110affd700 1 -- 172.21.15.156:0/1318895090 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v83) ==== 56+0+5360 (secure 0 0 0) 0x7f110c0ab760 con 0x7f1114104040 2024-05-26T12:03:26.257 INFO:teuthology.orchestra.run.smithi156.stdout:epoch 83 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:created 2024-05-26T11:52:07.537284+0000 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:modified 2024-05-26T12:02:08.534113+0000 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:crush_version 26 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:full_ratio 0.95 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:backfillfull_ratio 0.9 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:nearfull_ratio 0.85 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:require_min_compat_client luminous 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:min_compat_client luminous 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:require_osd_release squid 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:stretch_mode_enabled false 2024-05-26T12:03:26.258 INFO:teuthology.orchestra.run.smithi156.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 79 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:max_osd 12 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.0 up in weight 1 up_from 8 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6802/728684380,v1:172.21.15.156:6803/728684380] [v2:172.21.15.156:6804/728684380,v1:172.21.15.156:6805/728684380] exists,up e405c54e-bacd-4ac7-90d1-3969a01a02f1 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.1 up in weight 1 up_from 13 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6800/1098664623,v1:172.21.15.186:6801/1098664623] [v2:172.21.15.186:6802/1098664623,v1:172.21.15.186:6803/1098664623] exists,up 44233737-91d9-46b7-b9d5-fc82f5ac39ca 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.2 up in weight 1 up_from 18 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6800/4249467329,v1:172.21.15.192:6801/4249467329] [v2:172.21.15.192:6802/4249467329,v1:172.21.15.192:6803/4249467329] exists,up 50cc46b8-738c-4301-b70c-a20bbd1029fe 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.3 up in weight 1 up_from 26 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6810/2032209985,v1:172.21.15.156:6811/2032209985] [v2:172.21.15.156:6812/2032209985,v1:172.21.15.156:6813/2032209985] exists,up 31983ecf-dad2-4a4b-b72a-d24f0f8148ff 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.4 up in weight 1 up_from 32 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6808/1136966135,v1:172.21.15.186:6809/1136966135] [v2:172.21.15.186:6810/1136966135,v1:172.21.15.186:6811/1136966135] exists,up 3d2c072c-0fde-45be-b531-1179e7af010d 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.5 up in weight 1 up_from 37 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6808/199714532,v1:172.21.15.192:6809/199714532] [v2:172.21.15.192:6810/199714532,v1:172.21.15.192:6811/199714532] exists,up 43f06f8b-265a-4e50-92e9-01a5e2b28d59 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.6 up in weight 1 up_from 43 up_thru 76 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6818/4035023489,v1:172.21.15.156:6819/4035023489] [v2:172.21.15.156:6820/4035023489,v1:172.21.15.156:6821/4035023489] exists,up 7f4d15c5-136b-4322-8bd9-4069355ccafb 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.7 up in weight 1 up_from 48 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6816/2634988090,v1:172.21.15.186:6817/2634988090] [v2:172.21.15.186:6818/2634988090,v1:172.21.15.186:6819/2634988090] exists,up fe4f82a9-bb86-4323-b95f-69429a432679 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.8 up in weight 1 up_from 54 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6816/3218806357,v1:172.21.15.192:6817/3218806357] [v2:172.21.15.192:6818/3218806357,v1:172.21.15.192:6819/3218806357] exists,up 1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b 2024-05-26T12:03:26.259 INFO:teuthology.orchestra.run.smithi156.stdout:osd.9 up in weight 1 up_from 60 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.156:6826/1302726213,v1:172.21.15.156:6827/1302726213] [v2:172.21.15.156:6828/1302726213,v1:172.21.15.156:6829/1302726213] exists,up ea261f2e-8e96-430c-8e46-c1280b252e50 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:osd.10 up in weight 1 up_from 65 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.186:6824/714032142,v1:172.21.15.186:6825/714032142] [v2:172.21.15.186:6826/714032142,v1:172.21.15.186:6827/714032142] exists,up 30ce9960-8b0d-423a-b02b-5896204ddbf1 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:osd.11 up in weight 1 up_from 71 up_thru 80 down_at 0 last_clean_interval [0,0) [v2:172.21.15.192:6824/345136915,v1:172.21.15.192:6825/345136915] [v2:172.21.15.192:6826/345136915,v1:172.21.15.192:6827/345136915] exists,up 6330d810-016e-4e48-b4ca-4a71ae7317ff 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 2.6 [8,11] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 2.12 [5,11] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 2.14 [2,4] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 2.1a [8,1] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 2.1d [2,4,7,6] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 3.0 [4,6] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 3.7 [3,9] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 3.e [5,9] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 3.13 [4,9] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:pg_upmap_items 3.31 [4,8] 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:6801/2588638081 expires 2024-05-27T11:53:06.708196+0000 2024-05-26T12:03:26.260 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:6801/2544075824 expires 2024-05-27T11:52:40.563305+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/556773170 expires 2024-05-27T11:53:06.708196+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:6800/2544075824 expires 2024-05-27T11:52:40.563305+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/4114943616 expires 2024-05-27T11:52:40.563305+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/1606771671 expires 2024-05-27T11:53:06.708196+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/197533790 expires 2024-05-27T11:52:40.563305+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:6800/2588638081 expires 2024-05-27T11:53:06.708196+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/4033203939 expires 2024-05-27T11:52:40.563305+0000 2024-05-26T12:03:26.261 INFO:teuthology.orchestra.run.smithi156.stdout:blocklist 172.21.15.156:0/4071670553 expires 2024-05-27T11:53:06.708196+0000 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 msgr2=0x7f11000b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 0x7f11000b4d40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1110005950 tx=0x7f11100058e0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 msgr2=0x7f11141a0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f110c00d010 tx=0x7f110c007a20 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 shutdown_connections 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11000b2880 0x7f11000b4d40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1114104040 0x7f11141a0a50 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f11141024a0 0x7f111406bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 --2- 172.21.15.156:0/1318895090 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11141036a0 0x7f111419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.259+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 >> 172.21.15.156:0/1318895090 conn(0x7f11140fda40 msgr2=0x7f1114106a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:26.262 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.260+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 shutdown_connections 2024-05-26T12:03:26.263 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:26.260+0000 7f111bca3700 1 -- 172.21.15.156:0/1318895090 wait complete. 2024-05-26T12:03:26.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:26 smithi156 ceph-mon[32025]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:26.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:26 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1318895090' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-26T12:03:26.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:26 smithi186 ceph-mon[36412]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:26.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:26 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1318895090' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-26T12:03:26.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:26 smithi192 ceph-mon[36545]: pgmap v382: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:26.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:26 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1318895090' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-26T12:03:27.100 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-26T12:03:27.109 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.fs' 2024-05-26T12:03:27.349 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9b4e81fbc8cd4cc98feda2bf55b0f2ba Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 82, in run_one_task task = get_task(taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 35, in get_task module = _import('tasks', module_name, task_name, fail_on_import_error=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e7ae7580c417d6ab27fda98fecadca5d9f85d63e/teuthology/run_tasks.py", line 54, in _import module = __import__( ModuleNotFoundError: No module named 'tasks.fs' 2024-05-26T12:03:27.350 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-26T12:03:27.359 INFO:tasks.cephadm:Teardown begin 2024-05-26T12:03:27.359 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-26T12:03:27.360 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:03:27.608 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:28.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:28 smithi156 ceph-mon[32025]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:28.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:28 smithi186 ceph-mon[36412]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:28.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:28 smithi192 ceph-mon[36545]: pgmap v383: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:29.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 -- 172.21.15.156:0/992139523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01041a0 msgr2=0x7f6ac01081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:29.426 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 --2- 172.21.15.156:0/992139523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01041a0 0x7f6ac01081c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f6ab40099c0 tx=0x7f6ab4009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 -- 172.21.15.156:0/992139523 shutdown_connections 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 --2- 172.21.15.156:0/992139523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01088f0 0x7f6ac0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 --2- 172.21.15.156:0/992139523 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01037f0 0x7f6ac0103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 --2- 172.21.15.156:0/992139523 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01041a0 0x7f6ac01081c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 -- 172.21.15.156:0/992139523 >> 172.21.15.156:0/992139523 conn(0x7f6ac007a6a0 msgr2=0x7f6ac007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 -- 172.21.15.156:0/992139523 shutdown_connections 2024-05-26T12:03:29.427 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.423+0000 7f6ac574b700 1 -- 172.21.15.156:0/992139523 wait complete. 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 Processor -- start 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 -- start start 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01041a0 0x7f6ac019fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01088f0 0x7f6ac0199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac01137b0 con 0x7f6ac01037f0 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0113630 con 0x7f6ac01041a0 2024-05-26T12:03:29.428 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6ac574b700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0113930 con 0x7f6ac01088f0 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36682/0 (socket says 172.21.15.156:36682) 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 -- 172.21.15.156:0/822031739 learned_addr learned my addr 172.21.15.156:0/822031739 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abe7fc700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01041a0 0x7f6ac019fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abf7fe700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01088f0 0x7f6ac0199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 -- 172.21.15.156:0/822031739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01088f0 msgr2=0x7f6ac0199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01088f0 0x7f6ac0199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.429 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 -- 172.21.15.156:0/822031739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01041a0 msgr2=0x7f6ac019fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01041a0 0x7f6ac019fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.424+0000 7f6abeffd700 1 -- 172.21.15.156:0/822031739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac019a1d0 con 0x7f6ac01037f0 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6abeffd700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000b530 tx=0x7f6ab000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0015020 con 0x7f6ac01037f0 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac019a4c0 con 0x7f6ac01037f0 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ab0011ce0 con 0x7f6ac01037f0 2024-05-26T12:03:29.430 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0010dd0 con 0x7f6ac01037f0 2024-05-26T12:03:29.431 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.425+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac01a6800 con 0x7f6ac01037f0 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.426+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac010bb30 con 0x7f6ac01037f0 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.428+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f6ab0018020 con 0x7f6ac01037f0 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.428+0000 7f6aa7fff700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 0x7f6aa80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.428+0000 7f6abe7fc700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 0x7f6aa80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.429+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f6ab00e1f20 con 0x7f6ac01037f0 2024-05-26T12:03:29.432 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.429+0000 7f6abe7fc700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 0x7f6aa80b4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009990 tx=0x7f6ab40096d0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:29.433 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.432+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f6ab001f020 con 0x7f6ac01037f0 2024-05-26T12:03:29.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.678+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6ac006a230 con 0x7f6ac01037f0 2024-05-26T12:03:29.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.679+0000 7f6aa7fff700 1 -- 172.21.15.156:0/822031739 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7f6ab00a9c10 con 0x7f6ac01037f0 2024-05-26T12:03:29.681 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:29.681 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 msgr2=0x7f6aa80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 0x7f6aa80b4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009990 tx=0x7f6ab40096d0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 msgr2=0x7f6ac019f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000b530 tx=0x7f6ab000b840 comp rx=0 tx=0).stop 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 shutdown_connections 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f6aa80b2880 0x7f6aa80b4d40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f6ac01088f0 0x7f6ac0199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f6ac01037f0 0x7f6ac019f8b0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 --2- 172.21.15.156:0/822031739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac01041a0 0x7f6ac019fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:29.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 >> 172.21.15.156:0/822031739 conn(0x7f6ac007a6a0 msgr2=0x7f6ac01018a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:29.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 shutdown_connections 2024-05-26T12:03:29.684 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:29.682+0000 7f6ac574b700 1 -- 172.21.15.156:0/822031739 wait complete. 2024-05-26T12:03:29.685 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:03:30.477 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs dump --format=json 2024-05-26T12:03:30.721 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:30 smithi186 ceph-mon[36412]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:30.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:30 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/822031739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:30.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:30 smithi156 ceph-mon[32025]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:30.795 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:30 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/822031739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:30.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:30 smithi192 ceph-mon[36545]: pgmap v384: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-05-26T12:03:30.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:30 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/822031739' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:32.609 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- 172.21.15.156:0/3400415199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 msgr2=0x7f86f01081c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:32.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 --2- 172.21.15.156:0/3400415199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f01081c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f86e40099c0 tx=0x7f86e4009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- 172.21.15.156:0/3400415199 shutdown_connections 2024-05-26T12:03:32.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 --2- 172.21.15.156:0/3400415199 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01088f0 0x7f86f0110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 --2- 172.21.15.156:0/3400415199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01037f0 0x7f86f0103bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.610 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 --2- 172.21.15.156:0/3400415199 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f01081c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- 172.21.15.156:0/3400415199 >> 172.21.15.156:0/3400415199 conn(0x7f86f007a6a0 msgr2=0x7f86f007aab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- 172.21.15.156:0/3400415199 shutdown_connections 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- 172.21.15.156:0/3400415199 wait complete. 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 Processor -- start 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.607+0000 7f86f5478700 1 -- start start 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01037f0 0x7f86f0075840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 0x7f86f00762c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:32.611 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f86f01136b0 con 0x7f86f01041a0 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0113530 con 0x7f86f01088f0 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f86f0113830 con 0x7f86f01037f0 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:36700/0 (socket says 172.21.15.156:36700) 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 -- 172.21.15.156:0/4230583950 learned_addr learned my addr 172.21.15.156:0/4230583950 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86eeffd700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01037f0 0x7f86f0075840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:32.612 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ef7fe700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 0x7f86f00762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 -- 172.21.15.156:0/4230583950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01037f0 msgr2=0x7f86f0075840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01037f0 0x7f86f0075840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 -- 172.21.15.156:0/4230583950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 msgr2=0x7f86f00762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 0x7f86f00762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 -- 172.21.15.156:0/4230583950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f01a8040 con 0x7f86f01041a0 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ef7fe700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 0x7f86f00762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86ee7fc700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f86e400b580 tx=0x7f86e40047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4027070 con 0x7f86f01041a0 2024-05-26T12:03:32.613 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86f01a82d0 con 0x7f86f01041a0 2024-05-26T12:03:32.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86e4003f20 con 0x7f86f01041a0 2024-05-26T12:03:32.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.608+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86e4022d70 con 0x7f86f01041a0 2024-05-26T12:03:32.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.609+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86f01a8720 con 0x7f86f01041a0 2024-05-26T12:03:32.614 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.610+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86f006a230 con 0x7f86f01041a0 2024-05-26T12:03:32.615 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:32 smithi156 ceph-mon[32025]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:32.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.611+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f86e4022500 con 0x7f86f01041a0 2024-05-26T12:03:32.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.611+0000 7f86d7fff700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 0x7f86d80b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:32.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.612+0000 7f86eeffd700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 0x7f86d80b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:32.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.612+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f86e40e2f20 con 0x7f86f01041a0 2024-05-26T12:03:32.616 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.612+0000 7f86eeffd700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 0x7f86d80b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86e000ac50 tx=0x7f86e000a380 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:32.617 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.616+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f86e4013070 con 0x7f86f01041a0 2024-05-26T12:03:32.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:32 smithi186 ceph-mon[36412]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:32.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:32 smithi192 ceph-mon[36545]: pgmap v385: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-05-26T12:03:32.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.867+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f86f002e600 con 0x7f86f01041a0 2024-05-26T12:03:32.870 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.868+0000 7f86d7fff700 1 -- 172.21.15.156:0/4230583950 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11951 (secure 0 0 0) 0x7f86e40aaca0 con 0x7f86f01041a0 2024-05-26T12:03:32.871 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:32.872 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6835/1321300121","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6834","nonce":1321300121},{"type":"v1","addr":"172.21.15.156:6835","nonce":1321300121}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14547,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.156:6837/2020460154","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6836","nonce":2020460154},{"type":"v1","addr":"172.21.15.156:6837","nonce":2020460154}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-26T12:00:34.689787+0000","modified":"2024-05-26T12:00:55.501071+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24476,"mds_1":24458,"mds_2":24449,"mds_3":24442,"mds_4":24431},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14601":{"gid":14601,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6839/2826647780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6838","nonce":2826647780},{"type":"v1","addr":"172.21.15.156:6839","nonce":2826647780}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14613":{"gid":14613,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.156:6841/1148384159","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6840","nonce":1148384159},{"type":"v1","addr":"172.21.15.156:6841","nonce":1148384159}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24397":{"gid":24397,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6833/4217003952","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6832","nonce":4217003952},{"type":"v1","addr":"172.21.15.192:6833","nonce":4217003952}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24418":{"gid":24418,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6835/699682135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6834","nonce":699682135},{"type":"v1","addr":"172.21.15.192:6835","nonce":699682135}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24427":{"gid":24427,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.192:6837/977346748","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6836","nonce":977346748},{"type":"v1","addr":"172.21.15.192:6837","nonce":977346748}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24431":{"gid":24431,"name":"k","rank":4,"incarnation":25,"state":"up:active","state_seq":18,"addr":"172.21.15.186:6833/586672120","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":586672120},{"type":"v1","addr":"172.21.15.186:6833","nonce":586672120}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24442":{"gid":24442,"name":"i","rank":3,"incarnation":23,"state":"up:active","state_seq":12,"addr":"172.21.15.192:6839/1614885810","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6838","nonce":1614885810},{"type":"v1","addr":"172.21.15.192:6839","nonce":1614885810}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24449":{"gid":24449,"name":"e","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.186:6835/1012390034","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1012390034},{"type":"v1","addr":"172.21.15.186:6835","nonce":1012390034}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"h","rank":1,"incarnation":19,"state":"up:active","state_seq":13,"addr":"172.21.15.186:6837/3519203877","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6836","nonce":3519203877},{"type":"v1","addr":"172.21.15.186:6837","nonce":3519203877}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24476":{"gid":24476,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":6,"addr":"172.21.15.186:6839/2121675127","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6838","nonce":2121675127},{"type":"v1","addr":"172.21.15.186:6839","nonce":2121675127}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24476,"qdb_cluster":[24458,24476,24431,24442,24449]},"id":1}]} 2024-05-26T12:03:32.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 msgr2=0x7f86d80b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:32.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 0x7f86d80b4d40 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f86e000ac50 tx=0x7f86e000a380 comp rx=0 tx=0).stop 2024-05-26T12:03:32.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 msgr2=0x7f86f0075d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:32.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f86e400b580 tx=0x7f86e40047b0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.873 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 shutdown_connections 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f86d80b2880 0x7f86d80b4d40 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86f01088f0 0x7f86f00762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f86f01037f0 0x7f86f0075840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 --2- 172.21.15.156:0/4230583950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f86f01041a0 0x7f86f0075d80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.870+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 >> 172.21.15.156:0/4230583950 conn(0x7f86f007a6a0 msgr2=0x7f86f0109a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:32.874 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.871+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 shutdown_connections 2024-05-26T12:03:32.875 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:32.871+0000 7f86f5478700 1 -- 172.21.15.156:0/4230583950 wait complete. 2024-05-26T12:03:32.875 INFO:teuthology.orchestra.run.smithi156.stderr:dumped fsmap epoch 27 2024-05-26T12:03:33.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:33 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/4230583950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:33.663 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd dump --format=json 2024-05-26T12:03:33.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:33 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/4230583950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:33.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:33 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/4230583950' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-26T12:03:33.907 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:34.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:34 smithi186 ceph-mon[36412]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:34.794 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:34 smithi156 ceph-mon[32025]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:34.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:34 smithi192 ceph-mon[36545]: pgmap v386: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:35.673 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 -- 172.21.15.156:0/3455039285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c102df0 msgr2=0x7f002c10f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:35.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 --2- 172.21.15.156:0/3455039285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c102df0 0x7f002c10f2e0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f002400b0a0 tx=0x7f002400b3b0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 -- 172.21.15.156:0/3455039285 shutdown_connections 2024-05-26T12:03:35.674 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 --2- 172.21.15.156:0/3455039285 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c102df0 0x7f002c10f2e0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 --2- 172.21.15.156:0/3455039285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c108510 0x7f002c1088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 --2- 172.21.15.156:0/3455039285 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 0x7f002c1028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 -- 172.21.15.156:0/3455039285 >> 172.21.15.156:0/3455039285 conn(0x7f002c0793b0 msgr2=0x7f002c0797c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:35.675 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 -- 172.21.15.156:0/3455039285 shutdown_connections 2024-05-26T12:03:35.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 -- 172.21.15.156:0/3455039285 wait complete. 2024-05-26T12:03:35.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.669+0000 7f0031304700 1 Processor -- start 2024-05-26T12:03:35.676 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 -- start start 2024-05-26T12:03:35.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 0x7f002c104070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:35.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 0x7f002c1045b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:35.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:35.677 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1125e0 con 0x7f002c108510 2024-05-26T12:03:35.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f002c112460 con 0x7f002c102df0 2024-05-26T12:03:35.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f0031304700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f002c112760 con 0x7f002c102430 2024-05-26T12:03:35.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:35.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:33444/0 (socket says 172.21.15.156:33444) 2024-05-26T12:03:35.678 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 -- 172.21.15.156:0/1872833950 learned_addr learned my addr 172.21.15.156:0/1872833950 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:35.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002a7fc700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 0x7f002c1045b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:35.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002affd700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 0x7f002c104070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:35.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 -- 172.21.15.156:0/1872833950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 msgr2=0x7f002c104070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:35.679 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 0x7f002c104070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 -- 172.21.15.156:0/1872833950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 msgr2=0x7f002c1045b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 0x7f002c1045b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 -- 172.21.15.156:0/1872833950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c1a80c0 con 0x7f002c108510 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002a7fc700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 0x7f002c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.670+0000 7f002b7fe700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f002400b830 tx=0x7f0024004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.671+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024021070 con 0x7f002c108510 2024-05-26T12:03:35.680 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.671+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002c1a8350 con 0x7f002c108510 2024-05-26T12:03:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.671+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0024003c00 con 0x7f002c108510 2024-05-26T12:03:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.671+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024004730 con 0x7f002c108510 2024-05-26T12:03:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.671+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002c1a8850 con 0x7f002c108510 2024-05-26T12:03:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.672+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f002c06a230 con 0x7f002c108510 2024-05-26T12:03:35.681 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.673+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f0024003db0 con 0x7f002c108510 2024-05-26T12:03:35.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.674+0000 7f0013fff700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 0x7f00140b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:35.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.674+0000 7f002affd700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 0x7f00140b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:35.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.674+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f002400f020 con 0x7f002c108510 2024-05-26T12:03:35.682 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.674+0000 7f002affd700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 0x7f00140b4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f001c005fd0 tx=0x7f001c005ee0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:35.683 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.677+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f002402a020 con 0x7f002c108510 2024-05-26T12:03:35.885 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.884+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f002c02e720 con 0x7f002c108510 2024-05-26T12:03:35.886 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.886+0000 7f0013fff700 1 -- 172.21.15.156:0/1872833950 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+22753 (secure 0 0 0) 0x7f00240abad0 con 0x7f002c108510 2024-05-26T12:03:35.887 INFO:teuthology.orchestra.run.smithi156.stdout: 2024-05-26T12:03:35.887 INFO:teuthology.orchestra.run.smithi156.stdout:{"epoch":83,"fsid":"38c1aff8-1b56-11ef-bc9b-c7b262605968","created":"2024-05-26T11:52:07.537284+0000","modified":"2024-05-26T12:02:08.534113+0000","last_up_change":"2024-05-26T11:59:27.545679+0000","last_in_change":"2024-05-26T11:59:06.896167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-26T11:55:34.798357+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-26T12:00:26.642228+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-26T12:00:30.594873+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"79","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"e405c54e-bacd-4ac7-90d1-3969a01a02f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6802","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6803","nonce":728684380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6804","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6805","nonce":728684380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6808","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6809","nonce":728684380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6806","nonce":728684380},{"type":"v1","addr":"172.21.15.156:6807","nonce":728684380}]},"public_addr":"172.21.15.156:6803/728684380","cluster_addr":"172.21.15.156:6805/728684380","heartbeat_back_addr":"172.21.15.156:6809/728684380","heartbeat_front_addr":"172.21.15.156:6807/728684380","state":["exists","up"]},{"osd":1,"uuid":"44233737-91d9-46b7-b9d5-fc82f5ac39ca","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6801","nonce":1098664623}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6803","nonce":1098664623}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6807","nonce":1098664623}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":1098664623},{"type":"v1","addr":"172.21.15.186:6805","nonce":1098664623}]},"public_addr":"172.21.15.186:6801/1098664623","cluster_addr":"172.21.15.186:6803/1098664623","heartbeat_back_addr":"172.21.15.186:6807/1098664623","heartbeat_front_addr":"172.21.15.186:6805/1098664623","state":["exists","up"]},{"osd":2,"uuid":"50cc46b8-738c-4301-b70c-a20bbd1029fe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6800","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6801","nonce":4249467329}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6802","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6803","nonce":4249467329}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6806","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6807","nonce":4249467329}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6804","nonce":4249467329},{"type":"v1","addr":"172.21.15.192:6805","nonce":4249467329}]},"public_addr":"172.21.15.192:6801/4249467329","cluster_addr":"172.21.15.192:6803/4249467329","heartbeat_back_addr":"172.21.15.192:6807/4249467329","heartbeat_front_addr":"172.21.15.192:6805/4249467329","state":["exists","up"]},{"osd":3,"uuid":"31983ecf-dad2-4a4b-b72a-d24f0f8148ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6810","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6811","nonce":2032209985}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6812","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6813","nonce":2032209985}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6816","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6817","nonce":2032209985}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6814","nonce":2032209985},{"type":"v1","addr":"172.21.15.156:6815","nonce":2032209985}]},"public_addr":"172.21.15.156:6811/2032209985","cluster_addr":"172.21.15.156:6813/2032209985","heartbeat_back_addr":"172.21.15.156:6817/2032209985","heartbeat_front_addr":"172.21.15.156:6815/2032209985","state":["exists","up"]},{"osd":4,"uuid":"3d2c072c-0fde-45be-b531-1179e7af010d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6809","nonce":1136966135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6811","nonce":1136966135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6815","nonce":1136966135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":1136966135},{"type":"v1","addr":"172.21.15.186:6813","nonce":1136966135}]},"public_addr":"172.21.15.186:6809/1136966135","cluster_addr":"172.21.15.186:6811/1136966135","heartbeat_back_addr":"172.21.15.186:6815/1136966135","heartbeat_front_addr":"172.21.15.186:6813/1136966135","state":["exists","up"]},{"osd":5,"uuid":"43f06f8b-265a-4e50-92e9-01a5e2b28d59","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":37,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6808","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6809","nonce":199714532}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6810","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6811","nonce":199714532}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6814","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6815","nonce":199714532}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6812","nonce":199714532},{"type":"v1","addr":"172.21.15.192:6813","nonce":199714532}]},"public_addr":"172.21.15.192:6809/199714532","cluster_addr":"172.21.15.192:6811/199714532","heartbeat_back_addr":"172.21.15.192:6815/199714532","heartbeat_front_addr":"172.21.15.192:6813/199714532","state":["exists","up"]},{"osd":6,"uuid":"7f4d15c5-136b-4322-8bd9-4069355ccafb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6818","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6819","nonce":4035023489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6820","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6821","nonce":4035023489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6824","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6825","nonce":4035023489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6822","nonce":4035023489},{"type":"v1","addr":"172.21.15.156:6823","nonce":4035023489}]},"public_addr":"172.21.15.156:6819/4035023489","cluster_addr":"172.21.15.156:6821/4035023489","heartbeat_back_addr":"172.21.15.156:6825/4035023489","heartbeat_front_addr":"172.21.15.156:6823/4035023489","state":["exists","up"]},{"osd":7,"uuid":"fe4f82a9-bb86-4323-b95f-69429a432679","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6817","nonce":2634988090}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6819","nonce":2634988090}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6823","nonce":2634988090}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":2634988090},{"type":"v1","addr":"172.21.15.186:6821","nonce":2634988090}]},"public_addr":"172.21.15.186:6817/2634988090","cluster_addr":"172.21.15.186:6819/2634988090","heartbeat_back_addr":"172.21.15.186:6823/2634988090","heartbeat_front_addr":"172.21.15.186:6821/2634988090","state":["exists","up"]},{"osd":8,"uuid":"1fbcaa7a-8a32-4b65-9cb1-d6626f467e0b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6816","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6817","nonce":3218806357}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6818","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6819","nonce":3218806357}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6822","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6823","nonce":3218806357}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6820","nonce":3218806357},{"type":"v1","addr":"172.21.15.192:6821","nonce":3218806357}]},"public_addr":"172.21.15.192:6817/3218806357","cluster_addr":"172.21.15.192:6819/3218806357","heartbeat_back_addr":"172.21.15.192:6823/3218806357","heartbeat_front_addr":"172.21.15.192:6821/3218806357","state":["exists","up"]},{"osd":9,"uuid":"ea261f2e-8e96-430c-8e46-c1280b252e50","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6826","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6827","nonce":1302726213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6828","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6829","nonce":1302726213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6832","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6833","nonce":1302726213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.156:6830","nonce":1302726213},{"type":"v1","addr":"172.21.15.156:6831","nonce":1302726213}]},"public_addr":"172.21.15.156:6827/1302726213","cluster_addr":"172.21.15.156:6829/1302726213","heartbeat_back_addr":"172.21.15.156:6833/1302726213","heartbeat_front_addr":"172.21.15.156:6831/1302726213","state":["exists","up"]},{"osd":10,"uuid":"30ce9960-8b0d-423a-b02b-5896204ddbf1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6825","nonce":714032142}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6827","nonce":714032142}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6831","nonce":714032142}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":714032142},{"type":"v1","addr":"172.21.15.186:6829","nonce":714032142}]},"public_addr":"172.21.15.186:6825/714032142","cluster_addr":"172.21.15.186:6827/714032142","heartbeat_back_addr":"172.21.15.186:6831/714032142","heartbeat_front_addr":"172.21.15.186:6829/714032142","state":["exists","up"]},{"osd":11,"uuid":"6330d810-016e-4e48-b4ca-4a71ae7317ff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6824","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6825","nonce":345136915}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6826","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6827","nonce":345136915}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6830","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6831","nonce":345136915}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.192:6828","nonce":345136915},{"type":"v1","addr":"172.21.15.192:6829","nonce":345136915}]},"public_addr":"172.21.15.192:6825/345136915","cluster_addr":"172.21.15.192:6827/345136915","heartbeat_back_addr":"172.21.15.192:6831/345136915","heartbeat_front_addr":"172.21.15.192:6829/345136915","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:54:43.714139+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:07.600649+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:31.424656+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:55:56.570784+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:22.225523+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:56:47.534768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:13.945749+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:57:39.964925+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:05.441382+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:32.276086+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:58:58.647907+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-26T11:59:25.517919+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.12","mappings":[{"from":5,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4},{"from":7,"to":6}]},{"pgid":"3.0","mappings":[{"from":4,"to":6}]},{"pgid":"3.7","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.156:6801/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6801/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/556773170":"2024-05-27T11:53:06.708196+0000","172.21.15.156:6800/2544075824":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4114943616":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/1606771671":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/197533790":"2024-05-27T11:52:40.563305+0000","172.21.15.156:6800/2588638081":"2024-05-27T11:53:06.708196+0000","172.21.15.156:0/4033203939":"2024-05-27T11:52:40.563305+0000","172.21.15.156:0/4071670553":"2024-05-27T11:53:06.708196+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 msgr2=0x7f00140b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 0x7f00140b4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f001c005fd0 tx=0x7f001c005ee0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 msgr2=0x7f002c104af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f002400b830 tx=0x7f0024004d90 comp rx=0 tx=0).stop 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 shutdown_connections 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f00140b2880 0x7f00140b4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f002c108510 0x7f002c104af0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.889 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f002c102430 0x7f002c104070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 --2- 172.21.15.156:0/1872833950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f002c102df0 0x7f002c1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:35.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 >> 172.21.15.156:0/1872833950 conn(0x7f002c0793b0 msgr2=0x7f002c06be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:35.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 shutdown_connections 2024-05-26T12:03:35.890 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:35.888+0000 7f0031304700 1 -- 172.21.15.156:0/1872833950 wait complete. 2024-05-26T12:03:36.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:36 smithi156 ceph-mon[32025]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:36.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:36 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/1872833950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:03:36.610 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs fail cephfs 2024-05-26T12:03:36.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:36 smithi186 ceph-mon[36412]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:36.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:36 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/1872833950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:03:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:36 smithi192 ceph-mon[36545]: pgmap v387: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-05-26T12:03:36.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:36 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/1872833950' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-26T12:03:36.859 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:38.560 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-mon[32025]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:38.560 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 -- 172.21.15.156:0/2854813736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c109520 msgr2=0x7f119c109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 --2- 172.21.15.156:0/2854813736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c109520 0x7f119c109900 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f11980099c0 tx=0x7f1198009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 -- 172.21.15.156:0/2854813736 shutdown_connections 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 --2- 172.21.15.156:0/2854813736 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c1103d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 --2- 172.21.15.156:0/2854813736 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c109520 0x7f119c109900 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 --2- 172.21.15.156:0/2854813736 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c101b30 0x7f119c101fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.561 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 -- 172.21.15.156:0/2854813736 >> 172.21.15.156:0/2854813736 conn(0x7f119c078ab0 msgr2=0x7f119c0ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 -- 172.21.15.156:0/2854813736 shutdown_connections 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.559+0000 7f11a460a700 1 -- 172.21.15.156:0/2854813736 wait complete. 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 Processor -- start 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 -- start start 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c109520 0x7f119c1a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:38.562 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f119c113720 con 0x7f119c101b30 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f119c1135a0 con 0x7f119c109520 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a460a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f119c1138a0 con 0x7f119c1024f0 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a23a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a23a6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:33458/0 (socket says 172.21.15.156:33458) 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a23a6700 1 -- 172.21.15.156:0/3286231016 learned_addr learned my addr 172.21.15.156:0/3286231016 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a1ba5700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:38.563 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a2ba7700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c109520 0x7f119c1a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:38.564 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a1ba5700 1 -- 172.21.15.156:0/3286231016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c109520 msgr2=0x7f119c1a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a1ba5700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c109520 0x7f119c1a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.560+0000 7f11a1ba5700 1 -- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 msgr2=0x7f119c19bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a1ba5700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a1ba5700 1 -- 172.21.15.156:0/3286231016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c1a0e00 con 0x7f119c1024f0 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a23a6700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a1ba5700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c19c330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f118c00d950 tx=0x7f118c00dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c016070 con 0x7f119c1024f0 2024-05-26T12:03:38.565 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f118c00fd40 con 0x7f119c1024f0 2024-05-26T12:03:38.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118c00e3b0 con 0x7f119c1024f0 2024-05-26T12:03:38.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f119c1a10f0 con 0x7f119c1024f0 2024-05-26T12:03:38.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.561+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f119c10a2f0 con 0x7f119c1024f0 2024-05-26T12:03:38.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.562+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f119c06a230 con 0x7f119c1024f0 2024-05-26T12:03:38.566 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.563+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f118c019070 con 0x7f119c1024f0 2024-05-26T12:03:38.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.564+0000 7f11937fe700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 0x7f11880b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:38.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.565+0000 7f11a23a6700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 0x7f11880b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:38.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.565+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 8277+0+0 (secure 0 0 0) 0x7f118c0e20d0 con 0x7f119c1024f0 2024-05-26T12:03:38.567 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.565+0000 7f11a23a6700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 0x7f11880b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1198003800 tx=0x7f1198022040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:38.568 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.568+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f118c0a9d60 con 0x7f119c1024f0 2024-05-26T12:03:38.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:38 smithi186 ceph-mon[36412]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:38.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:38 smithi192 ceph-mon[36545]: pgmap v388: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-05-26T12:03:38.826 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.826+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f119c10a550 con 0x7f119c1024f0 2024-05-26T12:03:38.835 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.834+0000 7f11937fe700 1 -- 172.21.15.156:0/3286231016 <== mon.1 v2:172.21.15.192:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v28) ==== 161+0+0 (secure 0 0 0) 0x7f118c0a9660 con 0x7f119c1024f0 2024-05-26T12:03:38.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 msgr2=0x7f11880b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:38.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 0x7f11880b4d40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1198003800 tx=0x7f1198022040 comp rx=0 tx=0).stop 2024-05-26T12:03:38.837 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 msgr2=0x7f119c19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c19c330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f118c00d950 tx=0x7f118c00dc60 comp rx=0 tx=0).stop 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 shutdown_connections 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f11880b2880 0x7f11880b4d40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f119c109520 0x7f119c1a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f119c101b30 0x7f119c19bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 --2- 172.21.15.156:0/3286231016 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f119c1024f0 0x7f119c19c330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:38.838 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 >> 172.21.15.156:0/3286231016 conn(0x7f119c078ab0 msgr2=0x7f119c100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:38.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 shutdown_connections 2024-05-26T12:03:38.839 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:38.837+0000 7f11a460a700 1 -- 172.21.15.156:0/3286231016 wait complete. 2024-05-26T12:03:38.839 INFO:teuthology.orchestra.run.smithi156.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-26T12:03:39.226 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.227 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.227 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:03:38 smithi186 ceph-mds[67538]: -- 172.21.15.186:0/1942255613 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5593f9923a40 con 0x5593f98f6800 2024-05-26T12:03:39.227 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: starting mds.b at 2024-05-26T12:03:39.228 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.228 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.229 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:03:38 smithi186 ceph-mds[66914]: -- 172.21.15.186:0/2464063224 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ccd4c87a40 con 0x55ccd4c5a800 2024-05-26T12:03:39.229 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: starting mds.e at 2024-05-26T12:03:39.229 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.230 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.230 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:03:38 smithi186 ceph-mds[67223]: -- 172.21.15.186:0/1027706182 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5628a2491a40 con 0x5628a2465000 2024-05-26T12:03:39.230 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: starting mds.h at 2024-05-26T12:03:39.231 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.231 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.231 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:03:38 smithi186 ceph-mds[66600]: -- 172.21.15.186:0/126348008 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55759e725a40 con 0x55759e6f8c00 2024-05-26T12:03:39.232 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:03:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: starting mds.k at 2024-05-26T12:03:39.294 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a[32021]: 2024-05-26T12:03:38.827+0000 7fe499e72700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-26T12:03:39.295 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.295 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.295 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:03:38 smithi156 ceph-mds[70584]: -- 172.21.15.156:0/4128242095 <== mon.0 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55845afbda40 con 0x55845af90c00 2024-05-26T12:03:39.295 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: starting mds.g at 2024-05-26T12:03:39.296 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a[71023]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.296 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a[71023]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.296 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-mds[71040]: -- 172.21.15.156:0/1758249713 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f723b59a40 con 0x55f723b2c800 2024-05-26T12:03:39.296 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:03:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a[71023]: starting mds.a at 2024-05-26T12:03:39.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:03:38 smithi192 ceph-mds[66204]: -- 172.21.15.192:0/4169807504 <== mon.0 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55689df45a40 con 0x55689df18c00 2024-05-26T12:03:39.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: starting mds.l at 2024-05-26T12:03:39.323 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.323 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.323 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:03:38 smithi192 ceph-mds[67115]: -- 172.21.15.192:0/4274406957 <== mon.2 v2:172.21.15.186:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5584e88f1a40 con 0x5584e88c4c00 2024-05-26T12:03:39.323 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: starting mds.i at 2024-05-26T12:03:39.323 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:03:38 smithi192 ceph-mds[66507]: -- 172.21.15.192:0/2454407143 <== mon.0 v2:172.21.15.156:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56005bc0ba40 con 0x56005bbde800 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: starting mds.f at 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c[66807]: ignoring --setuser ceph since I am not root 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c[66807]: ignoring --setgroup ceph since I am not root 2024-05-26T12:03:39.324 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:03:38 smithi192 ceph-mds[66811]: -- 172.21.15.192:0/797039305 <== mon.1 v2:172.21.15.192:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d48b211a40 con 0x55d48b1e5000 2024-05-26T12:03:39.325 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:03:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c[66807]: starting mds.c at 2024-05-26T12:03:39.575 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:03:39.575 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3286231016' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.575 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.575 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-26T12:03:39.575 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: osdmap e84: 12 total, 12 up, 12 in 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.576 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.577 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:39 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.578 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph fs rm cephfs --yes-i-really-mean-it 2024-05-26T12:03:39.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3286231016' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-26T12:03:39.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: osdmap e84: 12 total, 12 up, 12 in 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:39 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3286231016' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: osdmap e84: 12 total, 12 up, 12 in 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-05-26T12:03:39.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:39 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:39.824 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:40.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-26T12:03:40.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:03:40.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:03:40.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:40.545 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6832/3827524526,v1:172.21.15.186:6833/3827524526] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6838/1291609300,v1:172.21.15.156:6840/1291609300] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.156:6839/2629185939,v1:172.21.15.156:6841/2629185939] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6834/1084740146,v1:172.21.15.186:6835/1084740146] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6832/2889266835,v1:172.21.15.192:6836/2889266835] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6833/3281897843,v1:172.21.15.192:6835/3281897843] up:boot 2024-05-26T12:03:40.546 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6834/4199859731,v1:172.21.15.192:6837/4199859731] up:boot 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.192:6838/3186701320,v1:172.21.15.192:6839/3186701320] up:boot 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6836/1601637113,v1:172.21.15.186:6837/1601637113] up:boot 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: mds.? [v2:172.21.15.186:6838/2102731816,v1:172.21.15.186:6839/2102731816] up:boot 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:03:40.547 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T12:03:40.548 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:40 smithi156 ceph-mon[32025]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T12:03:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-26T12:03:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:03:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:03:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:40.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6832/3827524526,v1:172.21.15.186:6833/3827524526] up:boot 2024-05-26T12:03:40.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6838/1291609300,v1:172.21.15.156:6840/1291609300] up:boot 2024-05-26T12:03:40.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.156:6839/2629185939,v1:172.21.15.156:6841/2629185939] up:boot 2024-05-26T12:03:40.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6834/1084740146,v1:172.21.15.186:6835/1084740146] up:boot 2024-05-26T12:03:40.728 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6832/2889266835,v1:172.21.15.192:6836/2889266835] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6833/3281897843,v1:172.21.15.192:6835/3281897843] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6834/4199859731,v1:172.21.15.192:6837/4199859731] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.192:6838/3186701320,v1:172.21.15.192:6839/3186701320] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6836/1601637113,v1:172.21.15.186:6837/1601637113] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: mds.? [v2:172.21.15.186:6838/2102731816,v1:172.21.15.186:6839/2102731816] up:boot 2024-05-26T12:03:40.729 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-26T12:03:40.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T12:03:40.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:03:40.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:03:40.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:03:40.730 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T12:03:40.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:03:40.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:03:40.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:03:40.731 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T12:03:40.732 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:40 smithi186 ceph-mon[36412]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T12:03:40.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: pgmap v390: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6832/3827524526,v1:172.21.15.186:6833/3827524526] up:boot 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6838/1291609300,v1:172.21.15.156:6840/1291609300] up:boot 2024-05-26T12:03:40.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.156:6839/2629185939,v1:172.21.15.156:6841/2629185939] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6834/1084740146,v1:172.21.15.186:6835/1084740146] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6832/2889266835,v1:172.21.15.192:6836/2889266835] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6833/3281897843,v1:172.21.15.192:6835/3281897843] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6834/4199859731,v1:172.21.15.192:6837/4199859731] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.192:6838/3186701320,v1:172.21.15.192:6839/3186701320] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6836/1601637113,v1:172.21.15.186:6837/1601637113] up:boot 2024-05-26T12:03:40.824 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: mds.? [v2:172.21.15.186:6838/2102731816,v1:172.21.15.186:6839/2102731816] up:boot 2024-05-26T12:03:40.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: fsmap cephfs:0/5 12 up:standby, 5 failed 2024-05-26T12:03:40.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-26T12:03:40.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-26T12:03:40.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-26T12:03:40.825 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-26T12:03:40.826 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:40 smithi192 ceph-mon[36545]: from='mgr.14150 172.21.15.156:0/1138730296' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-26T12:03:41.501 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 -- 172.21.15.156:0/136998521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60106ef0 msgr2=0x7fef6010eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:41.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 --2- 172.21.15.156:0/136998521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60106ef0 0x7fef6010eac0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fef5800b0d0 tx=0x7fef5800b3e0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 -- 172.21.15.156:0/136998521 shutdown_connections 2024-05-26T12:03:41.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 --2- 172.21.15.156:0/136998521 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60106ef0 0x7fef6010eac0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 --2- 172.21.15.156:0/136998521 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60101e60 0x7fef60102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.502 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 --2- 172.21.15.156:0/136998521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef60106810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 -- 172.21.15.156:0/136998521 >> 172.21.15.156:0/136998521 conn(0x7fef60078ab0 msgr2=0x7fef600ffe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 -- 172.21.15.156:0/136998521 shutdown_connections 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.499+0000 7fef6765a700 1 -- 172.21.15.156:0/136998521 wait complete. 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 Processor -- start 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 -- start start 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:41.503 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef6019af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 0x7fef601a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7fef60112620 con 0x7fef60101e60 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7fef601124a0 con 0x7fef60102810 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef6765a700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7fef601127a0 con 0x7fef60106ef0 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef653f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef653f6700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:33514/0 (socket says 172.21.15.156:33514) 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.500+0000 7fef653f6700 1 -- 172.21.15.156:0/95337124 learned_addr learned my addr 172.21.15.156:0/95337124 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:41.504 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef64bf5700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef6019af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef65bf7700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 0x7fef601a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 -- 172.21.15.156:0/95337124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 msgr2=0x7fef601a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 0x7fef601a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 -- 172.21.15.156:0/95337124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 msgr2=0x7fef6019af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef6019af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 -- 172.21.15.156:0/95337124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef601a5fa0 con 0x7fef60101e60 2024-05-26T12:03:41.505 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef64bf5700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef6019af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef65bf7700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 0x7fef601a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef653f6700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00c970 tx=0x7fef5c00cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5c007aa0 con 0x7fef60101e60 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef601a6170 con 0x7fef60101e60 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef5c004ce0 con 0x7fef60101e60 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef5c0055c0 con 0x7fef60101e60 2024-05-26T12:03:41.506 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.501+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef601a6680 con 0x7fef60101e60 2024-05-26T12:03:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.502+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef601032f0 con 0x7fef60101e60 2024-05-26T12:03:41.507 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.503+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7fef5c007c40 con 0x7fef60101e60 2024-05-26T12:03:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.504+0000 7fef567fc700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 0x7fef4c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.504+0000 7fef64bf5700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 0x7fef4c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.504+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8707+0+0 (secure 0 0 0) 0x7fef5c0e2500 con 0x7fef60101e60 2024-05-26T12:03:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.505+0000 7fef64bf5700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 0x7fef4c0b4d40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef60104ab0 tx=0x7fef50005db0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:41.508 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.507+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7fef5c0aa160 con 0x7fef60101e60 2024-05-26T12:03:41.759 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:41.758+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fef6002e560 con 0x7fef60101e60 2024-05-26T12:03:42.377 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.376+0000 7fef567fc700 1 -- 172.21.15.156:0/95337124 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v30) ==== 104+0+0 (secure 0 0 0) 0x7fef5c0a9a60 con 0x7fef60101e60 2024-05-26T12:03:42.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 msgr2=0x7fef4c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:42.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 0x7fef4c0b4d40 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fef60104ab0 tx=0x7fef50005db0 comp rx=0 tx=0).stop 2024-05-26T12:03:42.380 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 msgr2=0x7fef6019a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fef5c00c970 tx=0x7fef5c00cd30 comp rx=0 tx=0).stop 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 shutdown_connections 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7fef4c0b2880 0x7fef4c0b4d40 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7fef60106ef0 0x7fef601a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7fef60101e60 0x7fef6019a9e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 --2- 172.21.15.156:0/95337124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef60102810 0x7fef6019af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:42.381 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 >> 172.21.15.156:0/95337124 conn(0x7fef60078ab0 msgr2=0x7fef601055a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:42.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 shutdown_connections 2024-05-26T12:03:42.382 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:42.379+0000 7fef6765a700 1 -- 172.21.15.156:0/95337124 wait complete. 2024-05-26T12:03:42.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:42 smithi186 ceph-mon[36412]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:03:42.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:42 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T12:03:42.757 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:42 smithi156 ceph-mon[32025]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:03:42.758 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:42 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T12:03:42.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:42 smithi192 ceph-mon[36545]: pgmap v391: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-05-26T12:03:42.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:42 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-26T12:03:43.107 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-26T12:03:43.351 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:43.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:43 smithi186 ceph-mon[36412]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-26T12:03:43.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:43 smithi186 ceph-mon[36412]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-26T12:03:43.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:43 smithi186 ceph-mon[36412]: Cluster is now healthy 2024-05-26T12:03:43.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:43 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-26T12:03:43.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:43 smithi186 ceph-mon[36412]: fsmap 12 up:standby 2024-05-26T12:03:43.729 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:43 smithi156 ceph-mon[32025]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-26T12:03:43.729 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:43 smithi156 ceph-mon[32025]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-26T12:03:43.729 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:43 smithi156 ceph-mon[32025]: Cluster is now healthy 2024-05-26T12:03:43.729 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:43 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-26T12:03:43.729 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:43 smithi156 ceph-mon[32025]: fsmap 12 up:standby 2024-05-26T12:03:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:43 smithi192 ceph-mon[36545]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-26T12:03:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:43 smithi192 ceph-mon[36545]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-26T12:03:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:43 smithi192 ceph-mon[36545]: Cluster is now healthy 2024-05-26T12:03:43.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:43 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/95337124' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-26T12:03:43.823 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:43 smithi192 ceph-mon[36545]: fsmap 12 up:standby 2024-05-26T12:03:44.508 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:44 smithi156 ceph-mon[32025]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-26T12:03:44.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:44 smithi186 ceph-mon[36412]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-26T12:03:44.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:44 smithi192 ceph-mon[36545]: pgmap v392: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-05-26T12:03:45.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 -- 172.21.15.156:0/200277402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c101ee0 msgr2=0x7f8c4c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:45.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 --2- 172.21.15.156:0/200277402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c102360 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f8c400099c0 tx=0x7f8c40009cd0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.053 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 -- 172.21.15.156:0/200277402 shutdown_connections 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 --2- 172.21.15.156:0/200277402 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c110430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 --2- 172.21.15.156:0/200277402 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c108990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 --2- 172.21.15.156:0/200277402 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c102360 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 -- 172.21.15.156:0/200277402 >> 172.21.15.156:0/200277402 conn(0x7f8c4c0777e0 msgr2=0x7f8c4c077bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 -- 172.21.15.156:0/200277402 shutdown_connections 2024-05-26T12:03:45.054 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.052+0000 7f8c511ad700 1 -- 172.21.15.156:0/200277402 wait complete. 2024-05-26T12:03:45.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 Processor -- start 2024-05-26T12:03:45.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 -- start start 2024-05-26T12:03:45.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c1030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:45.056 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c103b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113790 con 0x7f8c4c1028a0 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113610 con 0x7f8c4c101ee0 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c511ad700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f8c4c113910 con 0x7f8c4c1085b0 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c4a59c700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:47590/0 (socket says 172.21.15.156:47590) 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.053+0000 7f8c4a59c700 1 -- 172.21.15.156:0/2289531743 learned_addr learned my addr 172.21.15.156:0/2289531743 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:45.057 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4b59e700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c103b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4ad9d700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c1030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 -- 172.21.15.156:0/2289531743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 msgr2=0x7f8c4c103b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c103b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 -- 172.21.15.156:0/2289531743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c101ee0 msgr2=0x7f8c4c1030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c1030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 -- 172.21.15.156:0/2289531743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c4c1a80f0 con 0x7f8c4c1028a0 2024-05-26T12:03:45.058 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4b59e700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c4a59c700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b580 tx=0x7f8c400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40027070 con 0x7f8c4c1028a0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c40004020 con 0x7f8c4c1028a0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c40022e30 con 0x7f8c4c1028a0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c4c1a8380 con 0x7f8c4c1028a0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.054+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c4c1a87d0 con 0x7f8c4c1028a0 2024-05-26T12:03:45.059 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.055+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c4c06b6b0 con 0x7f8c4c1028a0 2024-05-26T12:03:45.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.057+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f8c40022590 con 0x7f8c4c1028a0 2024-05-26T12:03:45.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.057+0000 7f8c33fff700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 0x7f8c340b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:45.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.057+0000 7f8c4ad9d700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 0x7f8c340b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:45.060 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.057+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 8707+0+0 (secure 0 0 0) 0x7f8c400e3510 con 0x7f8c4c1028a0 2024-05-26T12:03:45.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.058+0000 7f8c4ad9d700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 0x7f8c340b4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006fd0 tx=0x7f8c3c008040 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:45.061 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.061+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f8c40030070 con 0x7f8c4c1028a0 2024-05-26T12:03:45.264 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.263+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8c4c1a8b80 con 0x7f8c4c1028a0 2024-05-26T12:03:45.398 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.397+0000 7f8c33fff700 1 -- 172.21.15.156:0/2289531743 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) ==== 181+0+0 (secure 0 0 0) 0x7f8c40013070 con 0x7f8c4c1028a0 2024-05-26T12:03:45.401 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.400+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 msgr2=0x7f8c340b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.400+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 0x7f8c340b4d40 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8c3c006fd0 tx=0x7f8c3c008040 comp rx=0 tx=0).stop 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.400+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 msgr2=0x7f8c4c103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.400+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8c4000b580 tx=0x7f8c400047b0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 shutdown_connections 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f8c340b2880 0x7f8c340b4d40 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f8c4c1085b0 0x7f8c4c103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.402 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c4c101ee0 0x7f8c4c1030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 --2- 172.21.15.156:0/2289531743 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f8c4c1028a0 0x7f8c4c103600 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:45.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 >> 172.21.15.156:0/2289531743 conn(0x7f8c4c0777e0 msgr2=0x7f8c4c0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:45.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 shutdown_connections 2024-05-26T12:03:45.403 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:45.401+0000 7f8c511ad700 1 -- 172.21.15.156:0/2289531743 wait complete. 2024-05-26T12:03:45.403 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_metadata' removed 2024-05-26T12:03:45.544 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:45 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:45.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:45 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:45.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:45 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:46.098 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209 shell --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 -- ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-26T12:03:46.338 INFO:teuthology.orchestra.run.smithi156.stderr:Inferring config /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/mon.a/config 2024-05-26T12:03:46.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:46 smithi156 ceph-mon[32025]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-05-26T12:03:46.654 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:46 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:46.655 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:46 smithi156 ceph-mon[32025]: osdmap e85: 12 total, 12 up, 12 in 2024-05-26T12:03:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:46 smithi186 ceph-mon[36412]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-05-26T12:03:46.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:46 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:46.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:46 smithi186 ceph-mon[36412]: osdmap e85: 12 total, 12 up, 12 in 2024-05-26T12:03:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:46 smithi192 ceph-mon[36545]: pgmap v393: 97 pgs: 97 active+clean; 642 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 6 op/s 2024-05-26T12:03:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:46 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/2289531743' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:46.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:46 smithi192 ceph-mon[36545]: osdmap e85: 12 total, 12 up, 12 in 2024-05-26T12:03:48.079 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 -- 172.21.15.156:0/2627427786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60108520 msgr2=0x7f1f60108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/2627427786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60108520 0x7f1f60108900 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c009990 tx=0x7f1f5c009ca0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 -- 172.21.15.156:0/2627427786 shutdown_connections 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/2627427786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60102e00 0x7f1f6010f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/2627427786 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60108520 0x7f1f60108900 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/2627427786 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102440 0x7f1f601028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 -- 172.21.15.156:0/2627427786 >> 172.21.15.156:0/2627427786 conn(0x7f1f600793f0 msgr2=0x7f1f60079800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:48.080 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 -- 172.21.15.156:0/2627427786 shutdown_connections 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.077+0000 7f1f67bc5700 1 -- 172.21.15.156:0/2627427786 wait complete. 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 Processor -- start 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 -- start start 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 --2- >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 0x7f1f6019fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60108520 0x7f1f60199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 -- --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_getmap magic: 0 -- 0x7f1f601125f0 con 0x7f1f60102440 2024-05-26T12:03:48.081 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60112470 con 0x7f1f60108520 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f67bc5700 1 -- --> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] -- mon_getmap magic: 0 -- 0x7f1f60112770 con 0x7f1f60102e00 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 --2- >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.156:3300/0 says I am v2:172.21.15.156:47608/0 (socket says 172.21.15.156:47608) 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 -- 172.21.15.156:0/3736342884 learned_addr learned my addr 172.21.15.156:0/3736342884 (peer_addr_for_me v2:172.21.15.156:0/0) 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65160700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 0x7f1f6019fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f66162700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60108520 0x7f1f60199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:48.082 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 -- 172.21.15.156:0/3736342884 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 msgr2=0x7f1f6019fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 0x7f1f6019fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 -- 172.21.15.156:0/3736342884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60108520 msgr2=0x7f1f60199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60108520 0x7f1f60199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65961700 1 -- 172.21.15.156:0/3736342884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f6019a180 con 0x7f1f60102440 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.078+0000 7f1f65160700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 0x7f1f6019fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f65961700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0057f0 tx=0x7f1f5c003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c027070 con 0x7f1f60102440 2024-05-26T12:03:48.083 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1f5c0041c0 con 0x7f1f60102440 2024-05-26T12:03:48.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f6019a410 con 0x7f1f60102440 2024-05-26T12:03:48.084 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f5c022e60 con 0x7f1f60102440 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.079+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f601a6730 con 0x7f1f60102440 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.080+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f6006a230 con 0x7f1f60102440 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.081+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 4 ==== mgrmap(e 18) ==== 149653+0+0 (secure 0 0 0) 0x7f1f5c02a070 con 0x7f1f60102440 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.082+0000 7f1f56ffd700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 0x7f1f4c0b4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.082+0000 7f1f65160700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 0x7f1f4c0b4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-26T12:03:48.085 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.082+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8116+0+0 (secure 0 0 0) 0x7f1f5c0e3440 con 0x7f1f60102440 2024-05-26T12:03:48.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.082+0000 7f1f65160700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 0x7f1f4c0b4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f50005fd0 tx=0x7f1f50005dc0 comp rx=0 tx=0).ready entity=mgr.14150 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-26T12:03:48.086 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.085+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+191963 (secure 0 0 0) 0x7f1f5c013070 con 0x7f1f60102440 2024-05-26T12:03:48.287 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.286+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 --> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1f601a6960 con 0x7f1f60102440 2024-05-26T12:03:48.407 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.406+0000 7f1f56ffd700 1 -- 172.21.15.156:0/3736342884 <== mon.0 v2:172.21.15.156:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) ==== 169+0+0 (secure 0 0 0) 0x7f1f5c0ab2b0 con 0x7f1f60102440 2024-05-26T12:03:48.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.409+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 msgr2=0x7f1f4c0b4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:48.410 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.409+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 0x7f1f4c0b4d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1f50005fd0 tx=0x7f1f50005dc0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.409+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 msgr2=0x7f1f6019f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.409+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1f5c0057f0 tx=0x7f1f5c003730 comp rx=0 tx=0).stop 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 shutdown_connections 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:6800/3256012970,v1:172.21.15.156:6801/3256012970] conn(0x7f1f4c0b2880 0x7f1f4c0b4d40 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f60108520 0x7f1f60199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.156:3300/0,v1:172.21.15.156:6789/0] conn(0x7f1f60102440 0x7f1f6019f860 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.411 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 --2- 172.21.15.156:0/3736342884 >> [v2:172.21.15.192:3300/0,v1:172.21.15.192:6789/0] conn(0x7f1f60102e00 0x7f1f6019fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-26T12:03:48.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 >> 172.21.15.156:0/3736342884 conn(0x7f1f600793f0 msgr2=0x7f1f601016e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-26T12:03:48.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 shutdown_connections 2024-05-26T12:03:48.412 INFO:teuthology.orchestra.run.smithi156.stderr:2024-05-26T12:03:48.410+0000 7f1f67bc5700 1 -- 172.21.15.156:0/3736342884 wait complete. 2024-05-26T12:03:48.412 INFO:teuthology.orchestra.run.smithi156.stderr:pool 'cephfs_data' removed 2024-05-26T12:03:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:48 smithi186 ceph-mon[36412]: pgmap v395: 65 pgs: 65 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:03:48.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:48 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:48.751 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:48 smithi156 ceph-mon[32025]: pgmap v395: 65 pgs: 65 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:03:48.751 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:48 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:48 smithi192 ceph-mon[36545]: pgmap v395: 65 pgs: 65 active+clean; 577 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-26T12:03:48.822 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:48 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-26T12:03:49.098 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:03:49.127 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:03:49.160 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:03:49.191 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-26T12:03:49.192 DEBUG:teuthology.orchestra.run.smithi156:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-26T12:03:49.208 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-26T12:03:49.225 DEBUG:teuthology.orchestra.run.smithi192:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-26T12:03:49.250 INFO:tasks.cephadm:Stopping all daemons... 2024-05-26T12:03:49.251 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-26T12:03:49.251 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a 2024-05-26T12:03:49.412 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 systemd[1]: Stopping Ceph mon.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:49.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:49 smithi192 ceph-mon[36545]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:49.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:49 smithi192 ceph-mon[36545]: osdmap e86: 12 total, 12 up, 12 in 2024-05-26T12:03:49.726 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:49 smithi186 ceph-mon[36412]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:49.727 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:49 smithi186 ceph-mon[36412]: osdmap e86: 12 total, 12 up, 12 in 2024-05-26T12:03:49.781 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 ceph-mon[32025]: from='client.? 172.21.15.156:0/3736342884' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-26T12:03:49.781 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 ceph-mon[32025]: osdmap e86: 12 total, 12 up, 12 in 2024-05-26T12:03:49.781 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a[32021]: 2024-05-26T12:03:49.556+0000 7fe49f67d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:03:49.781 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a[32021]: 2024-05-26T12:03:49.556+0000 7fe49f67d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-26T12:03:50.046 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:49 smithi156 podman[85015]: 2024-05-26 12:03:49.781288727 +0000 UTC m=+0.337521087 container died 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:03:50.542 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:50 smithi156 podman[85015]: 2024-05-26 12:03:50.257865929 +0000 UTC m=+0.814098287 container cleanup 3cc8157ae7b03bb0a421c27b751acadba1f01710a050e9638c30cfa09845ca8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-26T12:03:50.543 INFO:journalctl@ceph.mon.a.smithi156.stdout:May 26 12:03:50 smithi156 bash[85015]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-a 2024-05-26T12:03:50.691 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.a.service' 2024-05-26T12:03:50.735 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:50.735 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-26T12:03:50.735 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-26T12:03:50.736 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.b 2024-05-26T12:03:51.048 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:50 smithi186 systemd[1]: Stopping Ceph mon.b for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:51.476 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-b[36408]: 2024-05-26T12:03:51.047+0000 7f779bed9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:03:51.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-b[36408]: 2024-05-26T12:03:51.047+0000 7f779bed9700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-26T12:03:51.477 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:51 smithi186 podman[69734]: 2024-05-26 12:03:51.283420048 +0000 UTC m=+0.345615946 container died 3991ed6389a20d4c9e6ba300b393fdca87826d3b82f45e2c55c2c62ae982d62b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0) 2024-05-26T12:03:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:51 smithi186 podman[69734]: 2024-05-26 12:03:51.775043008 +0000 UTC m=+0.837238902 container cleanup 3991ed6389a20d4c9e6ba300b393fdca87826d3b82f45e2c55c2c62ae982d62b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-b, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-05-26T12:03:51.976 INFO:journalctl@ceph.mon.b.smithi186.stdout:May 26 12:03:51 smithi186 bash[69734]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-b 2024-05-26T12:03:52.214 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.b.service' 2024-05-26T12:03:52.262 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:52.263 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-26T12:03:52.263 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-26T12:03:52.263 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.c 2024-05-26T12:03:52.572 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:52 smithi192 systemd[1]: Stopping Ceph mon.c for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:52.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-c[36541]: 2024-05-26T12:03:52.591+0000 7f407b2c7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:03:52.827 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-c[36541]: 2024-05-26T12:03:52.591+0000 7f407b2c7700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-26T12:03:53.110 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:52 smithi192 podman[69327]: 2024-05-26 12:03:52.827192198 +0000 UTC m=+0.371957958 container died a527f7e70c411534f73c19f0ce7a15a4c382e0ebb416f69b4ceea82f449047e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-26T12:03:53.570 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:53 smithi192 podman[69327]: 2024-05-26 12:03:53.302055561 +0000 UTC m=+0.846821324 container cleanup a527f7e70c411534f73c19f0ce7a15a4c382e0ebb416f69b4ceea82f449047e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-26T12:03:53.571 INFO:journalctl@ceph.mon.c.smithi192.stdout:May 26 12:03:53 smithi192 bash[69327]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mon-c 2024-05-26T12:03:53.739 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mon.c.service' 2024-05-26T12:03:53.783 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:53.783 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-26T12:03:53.784 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-26T12:03:53.784 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x 2024-05-26T12:03:54.087 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:03:53 smithi156 systemd[1]: Stopping Ceph mgr.x for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:54.439 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:03:54 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x[32254]: 2024-05-26T12:03:54.086+0000 7f71cfa1c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:03:54.439 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:03:54 smithi156 podman[85136]: 2024-05-26 12:03:54.119416116 +0000 UTC m=+0.143382301 container died 9bb67b950b96d10775e66c7bd583df7c88453d5ef62688389dbb71d37e7a749c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:03:54.773 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:03:54 smithi156 podman[85136]: 2024-05-26 12:03:54.538486854 +0000 UTC m=+0.562453029 container cleanup 9bb67b950b96d10775e66c7bd583df7c88453d5ef62688389dbb71d37e7a749c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9a4c5853d1c2a353f72cd6358bbdedd93c4cc209, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:03:54.773 INFO:journalctl@ceph.mgr.x.smithi156.stdout:May 26 12:03:54 smithi156 bash[85136]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-x 2024-05-26T12:03:54.989 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.x.service' 2024-05-26T12:03:55.024 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:55.025 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-26T12:03:55.025 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-26T12:03:55.025 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.y 2024-05-26T12:03:55.367 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 12:03:55 smithi186 systemd[1]: Stopping Ceph mgr.y for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:55.664 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 12:03:55 smithi186 podman[69856]: 2024-05-26 12:03:55.367299633 +0000 UTC m=+0.151854966 container died 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:03:55.976 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 12:03:55 smithi186 podman[69856]: 2024-05-26 12:03:55.848493835 +0000 UTC m=+0.633049177 container cleanup 05b15e8d722ffa8feacaa1135b367309a06c6e95017f571c20a8ca8f5feca618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-26T12:03:55.976 INFO:journalctl@ceph.mgr.y.smithi186.stdout:May 26 12:03:55 smithi186 bash[69856]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-y 2024-05-26T12:03:56.267 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.y.service' 2024-05-26T12:03:56.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:56.300 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-26T12:03:56.301 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-26T12:03:56.301 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.z 2024-05-26T12:03:56.645 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 12:03:56 smithi192 systemd[1]: Stopping Ceph mgr.z for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:56.950 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 12:03:56 smithi192 podman[69452]: 2024-05-26 12:03:56.644964301 +0000 UTC m=+0.150774181 container died 966d610e70fc22314956bcb7ffef4b62f91c84609101948f9f04ec48d32b4fd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-26T12:03:57.285 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 12:03:57 smithi192 podman[69452]: 2024-05-26 12:03:57.025178389 +0000 UTC m=+0.530988270 container cleanup 966d610e70fc22314956bcb7ffef4b62f91c84609101948f9f04ec48d32b4fd4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:03:57.285 INFO:journalctl@ceph.mgr.z.smithi192.stdout:May 26 12:03:57 smithi192 bash[69452]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mgr-z 2024-05-26T12:03:57.447 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mgr.z.service' 2024-05-26T12:03:57.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:03:57.480 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-26T12:03:57.481 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-26T12:03:57.481 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.0 2024-05-26T12:03:57.793 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:03:57 smithi156 systemd[1]: Stopping Ceph osd.0 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:03:58.044 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:03:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0[44160]: 2024-05-26T12:03:57.792+0000 7f4efff1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:03:58.045 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:03:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0[44160]: 2024-05-26T12:03:57.792+0000 7f4efff1f700 -1 osd.0 86 *** Got signal Terminated *** 2024-05-26T12:03:58.045 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:03:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0[44160]: 2024-05-26T12:03:57.792+0000 7f4efff1f700 -1 osd.0 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:03.503 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:03 smithi156 podman[85256]: 2024-05-26 12:04:03.213729118 +0000 UTC m=+5.538826505 container died d98985ed8b1359f1e2bd41b34b4be6d100f9ceed594d200fffaf7e567317f70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-05-26T12:04:03.794 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:03 smithi156 podman[85256]: 2024-05-26 12:04:03.593903441 +0000 UTC m=+5.919000822 container cleanup d98985ed8b1359f1e2bd41b34b4be6d100f9ceed594d200fffaf7e567317f70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:04:03.795 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:03 smithi156 bash[85256]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0 2024-05-26T12:04:04.223 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:03 smithi156 podman[85284]: 2024-05-26 12:04:03.853823419 +0000 UTC m=+0.637814557 container remove d98985ed8b1359f1e2bd41b34b4be6d100f9ceed594d200fffaf7e567317f70c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-05-26T12:04:04.545 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.126520938 +0000 UTC m=+0.017428409 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:04.545 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.254661262 +0000 UTC m=+0.145568721 container create c2b7f132e5d96f8fc504d51bb7003b33b4c191a16e30292632a30b71d051bc69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-05-26T12:04:04.883 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.586433868 +0000 UTC m=+0.477341344 container init c2b7f132e5d96f8fc504d51bb7003b33b4c191a16e30292632a30b71d051bc69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-26T12:04:04.883 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.592395357 +0000 UTC m=+0.483302823 container start c2b7f132e5d96f8fc504d51bb7003b33b4c191a16e30292632a30b71d051bc69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-26T12:04:04.884 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.757637891 +0000 UTC m=+0.648545361 container attach c2b7f132e5d96f8fc504d51bb7003b33b4c191a16e30292632a30b71d051bc69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True) 2024-05-26T12:04:05.244 INFO:journalctl@ceph.osd.0.smithi156.stdout:May 26 12:04:04 smithi156 podman[85371]: 2024-05-26 12:04:04.883924254 +0000 UTC m=+0.774831720 container died c2b7f132e5d96f8fc504d51bb7003b33b4c191a16e30292632a30b71d051bc69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:04:05.582 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.0.service' 2024-05-26T12:04:05.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:05.622 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-26T12:04:05.622 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-26T12:04:05.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.1 2024-05-26T12:04:05.976 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:05 smithi186 systemd[1]: Stopping Ceph osd.1 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:06.476 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1[40951]: 2024-05-26T12:04:06.053+0000 7f0db32b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:06.477 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1[40951]: 2024-05-26T12:04:06.053+0000 7f0db32b9700 -1 osd.1 86 *** Got signal Terminated *** 2024-05-26T12:04:06.477 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1[40951]: 2024-05-26T12:04:06.053+0000 7f0db32b9700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:11.905 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:11 smithi186 podman[69981]: 2024-05-26 12:04:11.598690639 +0000 UTC m=+5.781096079 container died 61df0fb6d7c7638135e62e15e41132b4fe1dec4e06b5759df1781f65242969ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:12.226 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[69981]: 2024-05-26 12:04:12.079915376 +0000 UTC m=+6.262320859 container cleanup 61df0fb6d7c7638135e62e15e41132b4fe1dec4e06b5759df1781f65242969ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-26T12:04:12.227 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 bash[69981]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1 2024-05-26T12:04:12.710 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[69997]: 2024-05-26 12:04:12.381713134 +0000 UTC m=+0.781145322 container remove 61df0fb6d7c7638135e62e15e41132b4fe1dec4e06b5759df1781f65242969ba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-26T12:04:12.976 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[70086]: 2024-05-26 12:04:12.61359553 +0000 UTC m=+0.017661347 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:12.977 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[70086]: 2024-05-26 12:04:12.73279281 +0000 UTC m=+0.136858617 container create a9cccfc15ed3586b26a2b327c9738a7b5f8f4591960bf857caf16b8bbbe450b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.build-date=20240521) 2024-05-26T12:04:12.977 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[70086]: 2024-05-26 12:04:12.943858569 +0000 UTC m=+0.347924378 container init a9cccfc15ed3586b26a2b327c9738a7b5f8f4591960bf857caf16b8bbbe450b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-26T12:04:12.977 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:12 smithi186 podman[70086]: 2024-05-26 12:04:12.946896459 +0000 UTC m=+0.350962268 container start a9cccfc15ed3586b26a2b327c9738a7b5f8f4591960bf857caf16b8bbbe450b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-26T12:04:13.237 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:13 smithi186 podman[70086]: 2024-05-26 12:04:13.05254146 +0000 UTC m=+0.456607259 container attach a9cccfc15ed3586b26a2b327c9738a7b5f8f4591960bf857caf16b8bbbe450b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5) 2024-05-26T12:04:13.513 INFO:journalctl@ceph.osd.1.smithi186.stdout:May 26 12:04:13 smithi186 podman[70086]: 2024-05-26 12:04:13.238423956 +0000 UTC m=+0.642489775 container died a9cccfc15ed3586b26a2b327c9738a7b5f8f4591960bf857caf16b8bbbe450b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521) 2024-05-26T12:04:13.929 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.1.service' 2024-05-26T12:04:13.967 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:13.967 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-26T12:04:13.967 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-26T12:04:13.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.2 2024-05-26T12:04:14.271 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:14 smithi192 systemd[1]: Stopping Ceph osd.2 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:14.572 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2[41782]: 2024-05-26T12:04:14.270+0000 7f074ae5f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:14.572 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2[41782]: 2024-05-26T12:04:14.270+0000 7f074ae5f700 -1 osd.2 86 *** Got signal Terminated *** 2024-05-26T12:04:14.572 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2[41782]: 2024-05-26T12:04:14.270+0000 7f074ae5f700 -1 osd.2 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:19.890 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:19 smithi192 podman[69574]: 2024-05-26 12:04:19.620842925 +0000 UTC m=+5.462563994 container died 5e82e4b68fd15e178e2889aae49f0c443626b7288bf37c10d6493521656c00ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5) 2024-05-26T12:04:20.322 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:20 smithi192 podman[69574]: 2024-05-26 12:04:20.073193203 +0000 UTC m=+5.914914274 container cleanup 5e82e4b68fd15e178e2889aae49f0c443626b7288bf37c10d6493521656c00ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521) 2024-05-26T12:04:20.322 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:20 smithi192 bash[69574]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2 2024-05-26T12:04:20.699 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:20 smithi192 podman[69593]: 2024-05-26 12:04:20.367127198 +0000 UTC m=+0.743398851 container remove 5e82e4b68fd15e178e2889aae49f0c443626b7288bf37c10d6493521656c00ca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:21.013 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:20 smithi192 podman[69680]: 2024-05-26 12:04:20.602021218 +0000 UTC m=+0.017361407 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:21.013 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:20 smithi192 podman[69680]: 2024-05-26 12:04:20.735022265 +0000 UTC m=+0.150362478 container create 8250e33b3022786ea59c99a44080b713149dd0d1e2dc02d6eab751dc5f19cbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-26T12:04:21.013 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:21 smithi192 podman[69680]: 2024-05-26 12:04:21.013246318 +0000 UTC m=+0.428586499 container init 8250e33b3022786ea59c99a44080b713149dd0d1e2dc02d6eab751dc5f19cbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2-deactivate, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:21.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:21 smithi192 podman[69680]: 2024-05-26 12:04:21.016913639 +0000 UTC m=+0.432253824 container start 8250e33b3022786ea59c99a44080b713149dd0d1e2dc02d6eab751dc5f19cbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:04:21.282 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:21 smithi192 podman[69680]: 2024-05-26 12:04:21.105044604 +0000 UTC m=+0.520384779 container attach 8250e33b3022786ea59c99a44080b713149dd0d1e2dc02d6eab751dc5f19cbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD) 2024-05-26T12:04:21.572 INFO:journalctl@ceph.osd.2.smithi192.stdout:May 26 12:04:21 smithi192 podman[69680]: 2024-05-26 12:04:21.318527429 +0000 UTC m=+0.733867612 container died 8250e33b3022786ea59c99a44080b713149dd0d1e2dc02d6eab751dc5f19cbd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-2-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:22.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.2.service' 2024-05-26T12:04:22.152 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:22.153 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-26T12:04:22.153 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-26T12:04:22.153 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.3 2024-05-26T12:04:22.464 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:22 smithi156 systemd[1]: Stopping Ceph osd.3 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:22.795 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:22 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:22.463+0000 7fc81f2b1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:22.795 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:22 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:22.463+0000 7fc81f2b1700 -1 osd.3 86 *** Got signal Terminated *** 2024-05-26T12:04:22.795 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:22 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:22.463+0000 7fc81f2b1700 -1 osd.3 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:23.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:23 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:23.261+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:24.521 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:24 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:24.338+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:24.545 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:24 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:24.356+0000 7fc81b8a5700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:01.890626+0000 front 2024-05-26T12:04:01.890642+0000 (oldest deadline 2024-05-26T12:04:24.091406+0000) 2024-05-26T12:04:24.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:24 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:24.264+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:25.544 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:25 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:25.397+0000 7fc81b8a5700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:01.890626+0000 front 2024-05-26T12:04:01.890642+0000 (oldest deadline 2024-05-26T12:04:24.091406+0000) 2024-05-26T12:04:25.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:25 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:25.285+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:25.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:25.285+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:25.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:25 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:25.356+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:26.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:26.258+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:26.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:26 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:26.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:26.794 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:26 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:26.351+0000 7fc81b8a5700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:01.890626+0000 front 2024-05-26T12:04:01.890642+0000 (oldest deadline 2024-05-26T12:04:24.091406+0000) 2024-05-26T12:04:26.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:26 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:26.315+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:26.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:26 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:26.379+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:27.297 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:26.945+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:27.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:27.297+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:27.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:27 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:27.348+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:27.794 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:27 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:27.429+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:27.795 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:27 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3[50676]: 2024-05-26T12:04:27.330+0000 7fc81b8a5700 -1 osd.3 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:01.890626+0000 front 2024-05-26T12:04:01.890642+0000 (oldest deadline 2024-05-26T12:04:24.091406+0000) 2024-05-26T12:04:27.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:27 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:27.308+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:28.248 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:27 smithi156 podman[85477]: 2024-05-26 12:04:27.947691357 +0000 UTC m=+5.596228024 container died 43f94096b45b7136b1a388c2f828d58cbcc0795ff96ec533672819c89764ef71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521) 2024-05-26T12:04:28.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:27.969+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:28.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:28 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:28.313+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:28.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:28 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:28.406+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:28.545 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:28 smithi156 podman[85477]: 2024-05-26 12:04:28.422940193 +0000 UTC m=+6.071476860 container cleanup 43f94096b45b7136b1a388c2f828d58cbcc0795ff96ec533672819c89764ef71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:28.545 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:28 smithi156 bash[85477]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3 2024-05-26T12:04:28.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:28 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:28.395+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:28.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:28.340+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:29.042 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:28 smithi156 podman[85494]: 2024-05-26 12:04:28.699413136 +0000 UTC m=+0.754083290 container remove 43f94096b45b7136b1a388c2f828d58cbcc0795ff96ec533672819c89764ef71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:04:29.294 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:28.945104547 +0000 UTC m=+0.017476647 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:29.295 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:29.067047784 +0000 UTC m=+0.139419875 container create 4996f11b62ae8370682a28a4f090906298649fea2d699f2842b25ec5940aaedb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-05-26T12:04:29.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:29.001+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:29.637 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:29 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:29.353+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:29.638 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:29 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:29.401+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:29.638 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:29.336266186 +0000 UTC m=+0.408638279 container init 4996f11b62ae8370682a28a4f090906298649fea2d699f2842b25ec5940aaedb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-26T12:04:29.638 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:29.33953323 +0000 UTC m=+0.411905334 container start 4996f11b62ae8370682a28a4f090906298649fea2d699f2842b25ec5940aaedb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-05-26T12:04:29.638 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:29.436364732 +0000 UTC m=+0.508736836 container attach 4996f11b62ae8370682a28a4f090906298649fea2d699f2842b25ec5940aaedb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-26T12:04:29.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:29 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:29.416+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:29.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:29.338+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:29.989 INFO:journalctl@ceph.osd.3.smithi156.stdout:May 26 12:04:29 smithi156 podman[85585]: 2024-05-26 12:04:29.638610756 +0000 UTC m=+0.710982880 container died 4996f11b62ae8370682a28a4f090906298649fea2d699f2842b25ec5940aaedb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0) 2024-05-26T12:04:30.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:29.987+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:30.421 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.3.service' 2024-05-26T12:04:30.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:30.468 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-26T12:04:30.468 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-26T12:04:30.469 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.4 2024-05-26T12:04:30.683 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:30 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:30.307+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:30.683 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:30 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:30.409+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:30.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:30 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:30.410+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:30.727 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:30 smithi186 systemd[1]: Stopping Ceph osd.4 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:30.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:30.337+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:31.226 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:30 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4[47844]: 2024-05-26T12:04:30.793+0000 7fd549f60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:31.226 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:30 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4[47844]: 2024-05-26T12:04:30.793+0000 7fd549f60700 -1 osd.4 86 *** Got signal Terminated *** 2024-05-26T12:04:31.226 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:30 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4[47844]: 2024-05-26T12:04:30.793+0000 7fd549f60700 -1 osd.4 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:31.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:30.975+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:31.613 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:31 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:31.297+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:31.613 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:31 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:31.399+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:31.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:31 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:31.453+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:31.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:31.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:32.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:31.976+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:32.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:32 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:32.149+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:32.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:32 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:32.427+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:32.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:32 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:32.335+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:32.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:32 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:32.351+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:32.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:32 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:32.351+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:32.821 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:32 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:32.416+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:33.247 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:32 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:32.928+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:33.247 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:32 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:32.928+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:33.410 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:33 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:33.119+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:33.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:33 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:33.402+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:33.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:33 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:33.402+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:33.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:33 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:33.409+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:33.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:33.323+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:33.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:33.323+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:33.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:33.312+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:33.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:33 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:33.312+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:34.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:33 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:33.941+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:34.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:33 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:33.941+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:34.388 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:34 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:34.134+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:34.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:34.283+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:34.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:34.283+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:34.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:34.321+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:34.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:34 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:34.321+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:34.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:34 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:34.388+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:34.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:34 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:34.388+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:34.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:34 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:34.568+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:34.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:34 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:34.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:34.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:34 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:34.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:35.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:34 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:34.913+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:35.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:34 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:34.913+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:35.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:35 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:35.180+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:35.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:35 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:35.394+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:35.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:35 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:35.394+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:35.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:35.305+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:35.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:35.305+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:35.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:35.358+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:35.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:35.358+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:35.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:35 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:35.527+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:35.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:35 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:35.495+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:35.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:35 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:35.495+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:36.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:35 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:35.914+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:36.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:35 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:35.914+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:36.396 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:36 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:36.130+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:36.396 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:36 smithi186 podman[70201]: 2024-05-26 12:04:36.191293307 +0000 UTC m=+5.531815072 container died de1c2d81fede27a4e0abca3bbc0c32e981c79566c1862a7c797ad08acca047b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:36.726 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:36 smithi186 podman[70201]: 2024-05-26 12:04:36.602707135 +0000 UTC m=+5.943228900 container cleanup de1c2d81fede27a4e0abca3bbc0c32e981c79566c1862a7c797ad08acca047b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:36.727 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:36 smithi186 bash[70201]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4 2024-05-26T12:04:36.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:36 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:36.395+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:36.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:36 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:36.395+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:36.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:36.347+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:36.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:36.347+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:36.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:36.318+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:36.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:36 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:36.318+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:36.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:36 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:36.558+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:36.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:36 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:36.462+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:36.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:36 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:36.462+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:37.174 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:36 smithi186 podman[70216]: 2024-05-26 12:04:36.854619707 +0000 UTC m=+0.660856808 container remove de1c2d81fede27a4e0abca3bbc0c32e981c79566c1862a7c797ad08acca047b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-26T12:04:37.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:36 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:36.894+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:37.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:36 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:36.894+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:37.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:37 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:37.173+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:37.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:37 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:37.354+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:37.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:37 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:37.354+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:37.477 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.125854834 +0000 UTC m=+0.017964121 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:37.477 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.247085959 +0000 UTC m=+0.139195223 container create cfa0ae9684db108cfc7404365ced4dfe277d0a3b8624d272fb936a38da10afe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5) 2024-05-26T12:04:37.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:37.391+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:37.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:37.391+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:37.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:37.290+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:37.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:37 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:37.290+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:37.821 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.522395394 +0000 UTC m=+0.414504662 container init cfa0ae9684db108cfc7404365ced4dfe277d0a3b8624d272fb936a38da10afe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_CLEAN=True) 2024-05-26T12:04:37.821 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.525406737 +0000 UTC m=+0.417516011 container start cfa0ae9684db108cfc7404365ced4dfe277d0a3b8624d272fb936a38da10afe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-26T12:04:37.821 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.608029484 +0000 UTC m=+0.500138775 container attach cfa0ae9684db108cfc7404365ced4dfe277d0a3b8624d272fb936a38da10afe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:37.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:37 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:37.582+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:37.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:37 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:37.505+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:37.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:37 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:37.505+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:38.129 INFO:journalctl@ceph.osd.4.smithi186.stdout:May 26 12:04:37 smithi186 podman[70306]: 2024-05-26 12:04:37.821818162 +0000 UTC m=+0.713927444 container died cfa0ae9684db108cfc7404365ced4dfe277d0a3b8624d272fb936a38da10afe9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-26T12:04:38.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:37 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:37.888+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:38.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:37 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:37.888+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:38.399 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:38.174+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:38.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.4.service' 2024-05-26T12:04:38.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:38.476 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-26T12:04:38.476 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-26T12:04:38.476 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.5 2024-05-26T12:04:38.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:38.360+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:38.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:38.360+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:38.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:38.246+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:38.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:38 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:38.246+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:38.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:38.398+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:38.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:38 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:38.398+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:38.812 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 systemd[1]: Stopping Ceph osd.5 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:38.813 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:38.562+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:38.530+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:38.813 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:38.530+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:39.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:38.812+0000 7f7bc09c6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:39.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:38.812+0000 7f7bc09c6700 -1 osd.5 86 *** Got signal Terminated *** 2024-05-26T12:04:39.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:38.812+0000 7f7bc09c6700 -1 osd.5 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:39.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:38.932+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:39.072 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:38 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:38.932+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:39.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:39 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:39.199+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:39.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:39 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:39.408+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:39.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:39 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:39.408+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:39.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:39.321+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:39.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:39.321+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:39.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:39.214+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:39.546 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:39 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:39.214+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:39.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:39 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:39.587+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:39.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:39 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:39.485+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:39.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:39 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:39.485+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:40.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:39 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:39.968+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:40.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:39 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:39.968+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:40.430 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:40 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:40.171+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:40.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:40.316+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:40.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:40.316+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:40.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:40.199+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:40.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:40.199+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:40.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:40 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:40.428+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:40.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:40 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:40.428+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:40.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:40 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:40.463+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:40.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:40 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:40.463+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:40.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:40 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:40.620+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:40.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:40 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:40.620+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:41.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:41.001+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:41.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:41.001+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:41.469 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:41 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:41.131+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:41.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:41 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:41.365+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:41.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:41 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:41.365+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:41.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:41 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:41.365+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:41.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:41 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:41.216+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:41.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:41 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:41.216+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:41.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:41 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:41.468+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:41.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:41 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:41.468+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:41.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:41.645+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:41.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:41.645+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:41.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:41.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:41.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:41.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:41.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:41.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:42.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:41.973+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:42.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:41 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:41.973+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:42.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:42 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:42.134+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:42.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:42 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:42.134+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:42.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:42 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:42.409+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:42.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:42 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:42.409+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:42.545 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:42 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:42.409+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:42.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:42 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:42.195+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:42.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:42 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:42.195+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:42.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:42.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:42.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:42.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:42.656 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:42.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:42.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:42 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:42.482+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:42.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:42 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:42.482+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:42.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:42 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:42.482+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:42.989 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:42.655+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:42.989 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:42.655+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:43.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:42.988+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:02.118919+0000 front 2024-05-26T12:04:02.118909+0000 (oldest deadline 2024-05-26T12:04:26.218802+0000) 2024-05-26T12:04:43.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:42.988+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.219246+0000 front 2024-05-26T12:04:06.219352+0000 (oldest deadline 2024-05-26T12:04:32.119240+0000) 2024-05-26T12:04:43.322 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:42 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5[48268]: 2024-05-26T12:04:42.988+0000 7f7bb8fb2700 -1 osd.5 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.920751+0000 front 2024-05-26T12:04:17.920781+0000 (oldest deadline 2024-05-26T12:04:42.620961+0000) 2024-05-26T12:04:43.447 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:43 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:43.148+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:43.447 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:43 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:43.148+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:43.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:43 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:43.104+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:43.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:43 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:43.104+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:43.700 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:43 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:43.423+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:43.700 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:43 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:43.423+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:43.700 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:43 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:43.423+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:43.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:43 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:43.446+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:43.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:43 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:43.446+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:43.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:43 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:43.446+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:43.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:43 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:43.485+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:43.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:43 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:43.485+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:43.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:43 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:43.485+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:44.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:43 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:43.699+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:44.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:43 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:43.699+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:44.424 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:44.126+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:44.425 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:44.126+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:44.425 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:44.126+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:44.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:44 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:44.087+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:44.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:44 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:44.087+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:44.488 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:44 smithi192 podman[69797]: 2024-05-26 12:04:44.215021698 +0000 UTC m=+5.536774989 container died 5fa9edc10d3910415802588c5201448334ee57e09f1beacdc3896dfab4d88f17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:44.488 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:44 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:44.457+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:44.488 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:44 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:44.457+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:44.488 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:44 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:44.457+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:44.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:44.424+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:44.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:44.424+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:44.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:44 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:44.424+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:44.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:44 smithi192 podman[69797]: 2024-05-26 12:04:44.695997873 +0000 UTC m=+6.017751164 container cleanup 5fa9edc10d3910415802588c5201448334ee57e09f1beacdc3896dfab4d88f17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-26T12:04:44.822 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:44 smithi192 bash[69797]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5 2024-05-26T12:04:44.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:44 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:44.684+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:44.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:44 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:44.684+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:44.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:44 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:44.536+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:44.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:44 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:44.536+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:44.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:44 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:44.536+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:45.306 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:44 smithi192 podman[69814]: 2024-05-26 12:04:44.989212594 +0000 UTC m=+0.772483341 container remove 5fa9edc10d3910415802588c5201448334ee57e09f1beacdc3896dfab4d88f17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-26T12:04:45.432 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:45.132+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:45.432 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:45.132+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:45.433 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:45.132+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:45.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:45.109+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:45.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:45.109+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:45.572 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.209866799 +0000 UTC m=+0.017194272 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:45.573 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.332015971 +0000 UTC m=+0.139343483 container create aa5ac6b2df3e0c03cc39700134d012b1e0c91e740e094a01e5a1f56b200eb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:04:45.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:45 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:45.435+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:45.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:45 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:45.435+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:45.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:45 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:45.435+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:45.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:45.432+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:45.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:45.432+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:45.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:45.432+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:45.901 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:45 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:45.664+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:45.901 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:45 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:45.664+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:45.901 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.60339334 +0000 UTC m=+0.410720815 container init aa5ac6b2df3e0c03cc39700134d012b1e0c91e740e094a01e5a1f56b200eb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:45.902 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.606758561 +0000 UTC m=+0.414086022 container start aa5ac6b2df3e0c03cc39700134d012b1e0c91e740e094a01e5a1f56b200eb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:04:45.902 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.692820782 +0000 UTC m=+0.500148245 container attach aa5ac6b2df3e0c03cc39700134d012b1e0c91e740e094a01e5a1f56b200eb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux ) 2024-05-26T12:04:45.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:45.530+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:45.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:45.530+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:45.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:45 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:45.530+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:46.254 INFO:journalctl@ceph.osd.5.smithi192.stdout:May 26 12:04:45 smithi192 podman[69902]: 2024-05-26 12:04:45.901624472 +0000 UTC m=+0.708951933 container died aa5ac6b2df3e0c03cc39700134d012b1e0c91e740e094a01e5a1f56b200eb829 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-5-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-26T12:04:46.469 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:46.114+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:46.469 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:46.114+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:46.470 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:46.114+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:46.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:46 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:46.079+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:46.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:46 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:46.079+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:46 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:46.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:46 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:46.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:46.563 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:46 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:46.428+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:46.581 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.5.service' 2024-05-26T12:04:46.619 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:46.619 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-26T12:04:46.619 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-26T12:04:46.619 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.6 2024-05-26T12:04:46.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.468+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:46.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.468+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:46.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.468+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:46.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 systemd[1]: Stopping Ceph osd.6 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:46.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:46 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:46.628+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:46.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:46 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:46.628+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:46.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:46 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:46.554+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:46.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:46 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:46.554+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:46.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:46 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:46.554+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:47.294 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.934+0000 7f3496522700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:47.295 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.934+0000 7f3496522700 -1 osd.6 86 *** Got signal Terminated *** 2024-05-26T12:04:47.295 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:46 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:46.934+0000 7f3496522700 -1 osd.6 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:47.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:47.090+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:47.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:47.090+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:47.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:47.090+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:47.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:47 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:47.111+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:47.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:47 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:47.111+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:47.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:47.474+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:47.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:47.474+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:47.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:47 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:47.474+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:47.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:47 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:47.593+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:47.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:47 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:47.593+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:47.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:47 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:47.382+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:47.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:47 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:47.382+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:47.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:47 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:47.382+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:47.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:47 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:47.537+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:47.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:47 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:47.537+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:47.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:47 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:47.537+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:48.428 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:48.128+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:48.428 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:48.128+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:48.428 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:48.128+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:48.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:48.088+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:48.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:48.088+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:48.633 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:48.361+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:48.634 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:48.361+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:48.634 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:48.361+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:48.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:48.428+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:48.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:48.428+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:48.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:48.428+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:48.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:48.515+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:48.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:48.515+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:48.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:48.515+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:48.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:48 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:48.515+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:49.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:48.633+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:49.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:48 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:48.633+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:49.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:49.112+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:49.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:49.112+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:49.478 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:49.173+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:49.478 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:49.173+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:49.478 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:49.173+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:49.478 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:49.173+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:49.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:49.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:49.658 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:49.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:49.659 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:49.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:49.659 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:49.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:49.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:49.476+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:49.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:49.476+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:49.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:49 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:49.476+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:49.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:49.531+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:49.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:49.531+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:49.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:49.531+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:49.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:49 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:49.531+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:50.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:49.657+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:50.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:49.657+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:50.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:49 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:49.657+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:50.450 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:50.166+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:50.450 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:50.166+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:50.450 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:50.166+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:50.450 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:50.166+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:50.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:50.100+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:50.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:50.100+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:50.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:50.100+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:50.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:50.449+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:50.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:50.449+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:50.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:50.449+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:50.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:50 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:50.449+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.757130+0000 front 2024-05-26T12:04:24.757158+0000 (oldest deadline 2024-05-26T12:04:50.057264+0000) 2024-05-26T12:04:50.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:50.635+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:50.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:50.635+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:50.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:50.635+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:50.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:50.397+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:50.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:50.397+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:50.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:50.397+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:50.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:50 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:50.397+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:50.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:50.493+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:50.976 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:50.493+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:50.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:50.493+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:50.977 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:50.493+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:51.463 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:51.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:51.464 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:51.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:51.464 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:51.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:51.499 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:51.133+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:51.499 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:51.133+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:51.499 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:51.133+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:51.500 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:51.133+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:51.654 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:51.350+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:51.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:51.350+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:51.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:51.350+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:51.655 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:51.350+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:51.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:51.463+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:51.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:51.463+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:51.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:51.463+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:51.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:51 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:51.463+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:51.794 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:51.498+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.152774+0000 front 2024-05-26T12:03:59.152806+0000 (oldest deadline 2024-05-26T12:04:23.253042+0000) 2024-05-26T12:04:51.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:51.498+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.554183+0000 front 2024-05-26T12:04:08.554231+0000 (oldest deadline 2024-05-26T12:04:33.254245+0000) 2024-05-26T12:04:51.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:51.498+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.855913+0000 front 2024-05-26T12:04:17.856032+0000 (oldest deadline 2024-05-26T12:04:41.355864+0000) 2024-05-26T12:04:51.795 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:51 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6[57198]: 2024-05-26T12:04:51.498+0000 7f3492b16700 -1 osd.6 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.757130+0000 front 2024-05-26T12:04:24.757158+0000 (oldest deadline 2024-05-26T12:04:50.057264+0000) 2024-05-26T12:04:52.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:51.653+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:52.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:51.653+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:52.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:51 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:51.653+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:52.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:52.180+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:52.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:52.180+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:52.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:52.180+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:52.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:52.421+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:52.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:52.421+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:52.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:52.421+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:52.478 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:52 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:52.421+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:52.544 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:52 smithi156 podman[85702]: 2024-05-26 12:04:52.378211806 +0000 UTC m=+5.563856296 container died f9967b0e01f5a8b8b0a890d96e6fb488b29d1622a858d3923e7cbe2200c57b92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:04:52.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:52 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:52.130+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:52.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:52 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:52.130+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:52.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:52 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:52.130+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:52.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:52 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:52.130+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:52.629 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:52.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:52.629 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:52.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:52.629 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:52.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:52.629 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:52.353+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:52.902 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:52.629+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:52.905 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:52.629+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:52.905 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:52 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:52.629+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:53.044 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:52 smithi156 podman[85702]: 2024-05-26 12:04:52.817893342 +0000 UTC m=+6.003537874 container cleanup f9967b0e01f5a8b8b0a890d96e6fb488b29d1622a858d3923e7cbe2200c57b92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:04:53.044 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:52 smithi156 bash[85702]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6 2024-05-26T12:04:53.294 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85720]: 2024-05-26 12:04:53.044562333 +0000 UTC m=+0.664897030 container remove f9967b0e01f5a8b8b0a890d96e6fb488b29d1622a858d3923e7cbe2200c57b92 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:04:53.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:53 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:53.159+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:53.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:53 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:53.159+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:53.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:53 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:53.159+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:53.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:53 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:53.159+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:53.415 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:53.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:53.416 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:53.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:53.416 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:53.138+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:53.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:53.415+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:53.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:53.415+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:53.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:53.415+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:53.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:53 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:53.415+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:53.730 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85806]: 2024-05-26 12:04:53.352827817 +0000 UTC m=+0.017194919 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:04:53.730 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85806]: 2024-05-26 12:04:53.478774978 +0000 UTC m=+0.143142064 container create e0cb4f2a36b08bddb1eaf14ddd10696e62f83092ee946bf65d0c4138af6d7824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux ) 2024-05-26T12:04:53.731 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85806]: 2024-05-26 12:04:53.731179765 +0000 UTC m=+0.395546859 container init e0cb4f2a36b08bddb1eaf14ddd10696e62f83092ee946bf65d0c4138af6d7824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:04:53.821 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:53.593+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:53.821 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:53.593+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:53.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:53.593+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:53.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:53.354+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:53.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:53.354+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:53.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:53.354+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:53.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:53 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:53.354+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:54.030 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85806]: 2024-05-26 12:04:53.734642003 +0000 UTC m=+0.399009089 container start e0cb4f2a36b08bddb1eaf14ddd10696e62f83092ee946bf65d0c4138af6d7824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-26T12:04:54.030 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:53 smithi156 podman[85806]: 2024-05-26 12:04:53.814600412 +0000 UTC m=+0.478967507 container attach e0cb4f2a36b08bddb1eaf14ddd10696e62f83092ee946bf65d0c4138af6d7824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521) 2024-05-26T12:04:54.295 INFO:journalctl@ceph.osd.6.smithi156.stdout:May 26 12:04:54 smithi156 podman[85806]: 2024-05-26 12:04:54.030859511 +0000 UTC m=+0.695226599 container died e0cb4f2a36b08bddb1eaf14ddd10696e62f83092ee946bf65d0c4138af6d7824 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD) 2024-05-26T12:04:54.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:54 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:54.160+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:54.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:54 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:54.160+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:54.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:54 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:54.160+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:54.296 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:54 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:54.160+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:54.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:54.179+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:54.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:54.179+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:54.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:54.179+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:54.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:54.420+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:54.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:54.420+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:54.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:54.420+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:54.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:54 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:54.420+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:54.820 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.6.service' 2024-05-26T12:04:54.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:54.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:54.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:54.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:54.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:54.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:54.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:54.377+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:54.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:54.377+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:54.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:54.377+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:54.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:54 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:54.377+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:54.858 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:04:54.858 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-26T12:04:54.859 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-26T12:04:54.859 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.7 2024-05-26T12:04:55.157 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:54 smithi186 systemd[1]: Stopping Ceph osd.7 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.163+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.163+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.163+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.189+0000 7f173743c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.189+0000 7f173743c700 -1 osd.7 86 *** Got signal Terminated *** 2024-05-26T12:04:55.427 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:55.189+0000 7f173743c700 -1 osd.7 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:04:55.544 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:55 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:55.174+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:55.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:55 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:55.174+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:55.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:55 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:55.174+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:55.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:55 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:55.174+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:55.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:55.426+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:55.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:55.426+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:55.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:55.426+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:55.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:55 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:55.426+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:55.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:55.548+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:55.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:55.548+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:55.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:55.548+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:55.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:55.362+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:55.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:55.362+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:55.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:55.362+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:55.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:55 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:55.362+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:56.387 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:56.209+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:56.387 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:56.209+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:56.387 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:56.209+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:56.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:56 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:56.149+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:56.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:56 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:56.149+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:56.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:56 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:56.149+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:56.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:56 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:56.149+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:56.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:56.386+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:56.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:56.386+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:56.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:56.386+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:56.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:56.386+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:56.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:56.522+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:56.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:56.522+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:56.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:56.522+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:56.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:56.366+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:56.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:56.366+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:56.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:56.366+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:56.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:56 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:56.366+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:57.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:57.208+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:57.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:57.208+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:57.477 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:57.208+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:57.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:57.361+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:57.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:57.361+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:57.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:57.361+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:57.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:57 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:57.361+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:57.544 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:57.100+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:57.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:57.100+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:57.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:57.100+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:57.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:57 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:57.100+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:57.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:57.521+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:57.521+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:57.521+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:57.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:57.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:57.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:57.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:57 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:57.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:58.446 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:58.060+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:58.446 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:58.060+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:58.447 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:58.060+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:58.447 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:58 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:58.060+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:58.726 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:58.234+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:58.234+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:58.234+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:58.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:58.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:58.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:58.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:58 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:58.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:58.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:58.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:58.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:58.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:58.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:58.551+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:58.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:58.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:58.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:58.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:58.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:58.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:58.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:58.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:58.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:58 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:58.398+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:04:59.294 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.014+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:04:59.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.014+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:04:59.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.014+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:04:59.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.014+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:04:59.726 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:59.240+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:06.703968+0000 front 2024-05-26T12:04:06.703943+0000 (oldest deadline 2024-05-26T12:04:32.004154+0000) 2024-05-26T12:04:59.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:59.240+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:16.605761+0000 front 2024-05-26T12:04:16.605787+0000 (oldest deadline 2024-05-26T12:04:41.305718+0000) 2024-05-26T12:04:59.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:59.240+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.107336+0000 front 2024-05-26T12:04:27.107470+0000 (oldest deadline 2024-05-26T12:04:49.407434+0000) 2024-05-26T12:04:59.727 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7[54848]: 2024-05-26T12:04:59.240+0000 7f1733a30700 -1 osd.7 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:32.809093+0000 front 2024-05-26T12:04:32.809116+0000 (oldest deadline 2024-05-26T12:04:58.709274+0000) 2024-05-26T12:04:59.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:59.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:04:59.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:59.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:04:59.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:59.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:04:59.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:59.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:04:59.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:04:59 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:04:59.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:04:59.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:59.528+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:04:59.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:59.528+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:04:59.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:04:59.528+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:04:59.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:59.392+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:04:59.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:59.392+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:04:59.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:59.392+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:04:59.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:59.392+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:04:59.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:04:59 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:04:59.392+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:00.294 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.996+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:00.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.996+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:00.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.996+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:00.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:04:59 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:04:59.996+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:00.726 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:00 smithi186 podman[70412]: 2024-05-26 12:05:00.515494699 +0000 UTC m=+5.467051994 container died 0eb8760bbfee906da69781cbba692003efc17be1bdab20b8b889464233dbd11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-05-26T12:05:00.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:00.370+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:00.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:00.370+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:00.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:00.370+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:00.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:00.370+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:00.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:00.370+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:00.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:00.536+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:00.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:00.536+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:00.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:00.536+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:00.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:00.360+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:00.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:00.360+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:00.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:00.360+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:00.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:00.360+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:00.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:00 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:00.360+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:01.199 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:00 smithi186 podman[70412]: 2024-05-26 12:05:00.964516329 +0000 UTC m=+5.916073623 container cleanup 0eb8760bbfee906da69781cbba692003efc17be1bdab20b8b889464233dbd11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:05:01.199 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:00 smithi186 bash[70412]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7 2024-05-26T12:05:01.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:01.011+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:01.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:01.011+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:01.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:01.011+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:01.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:01 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:01.011+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:01.476 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70438]: 2024-05-26 12:05:01.199588899 +0000 UTC m=+0.676596006 container remove 0eb8760bbfee906da69781cbba692003efc17be1bdab20b8b889464233dbd11f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-26T12:05:01.476 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:01 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:01.363+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:01.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:01 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:01.363+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:01.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:01 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:01.363+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:01.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:01 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:01.363+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:01.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:01 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:01.363+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:01.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:01.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:01.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:01.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:01.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:01.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:01.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:01.388+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:01.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:01.388+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:01.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:01.388+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:01.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:01.388+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:01.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:01 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:01.388+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:01.944 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70530]: 2024-05-26 12:05:01.536507326 +0000 UTC m=+0.017555768 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:05:01.944 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70530]: 2024-05-26 12:05:01.633829836 +0000 UTC m=+0.114878287 container create 73d1d403893e7674be67f84e5db2900a149863a6dcb3e3e4e2f84f311c34e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-26T12:05:01.945 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70530]: 2024-05-26 12:05:01.85268467 +0000 UTC m=+0.333733098 container init 73d1d403893e7674be67f84e5db2900a149863a6dcb3e3e4e2f84f311c34e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS) 2024-05-26T12:05:01.945 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70530]: 2024-05-26 12:05:01.855978807 +0000 UTC m=+0.337027238 container start 73d1d403893e7674be67f84e5db2900a149863a6dcb3e3e4e2f84f311c34e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:01.945 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:01 smithi186 podman[70530]: 2024-05-26 12:05:01.944347632 +0000 UTC m=+0.425396070 container attach 73d1d403893e7674be67f84e5db2900a149863a6dcb3e3e4e2f84f311c34e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521) 2024-05-26T12:05:02.226 INFO:journalctl@ceph.osd.7.smithi186.stdout:May 26 12:05:02 smithi186 podman[70530]: 2024-05-26 12:05:02.151589352 +0000 UTC m=+0.632637780 container died 73d1d403893e7674be67f84e5db2900a149863a6dcb3e3e4e2f84f311c34e3a3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-7-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:05:02.294 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:02.021+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:02.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:02.021+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:02.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:02.021+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:02.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:02 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:02.021+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:02.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:02 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:02.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:02.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:02 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:02.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:02.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:02 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:02.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:02.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:02 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:02.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:02.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:02 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:02.378+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:02.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:02.500+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:02.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:02.500+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:02.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:02.500+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:02.500+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:02.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:02.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:02.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:02.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:02.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:02 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:02.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:02.916 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.7.service' 2024-05-26T12:05:02.956 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:02.956 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-26T12:05:02.957 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-26T12:05:02.957 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.8 2024-05-26T12:05:03.274 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 systemd[1]: Stopping Ceph osd.8 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:03.274 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.273+0000 7fdee1edd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:03.275 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.273+0000 7fdee1edd700 -1 osd.8 86 *** Got signal Terminated *** 2024-05-26T12:05:03.275 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.273+0000 7fdee1edd700 -1 osd.8 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:05:03.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:03.066+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:03.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:03.066+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:03.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:03.066+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:03.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:03 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:03.066+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:03.572 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:03.572 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:03.572 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:03.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:03.411+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:03.411+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:03.411+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:03.411+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:03.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:03 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:03.411+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:03.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:03 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:03.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:03.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:03 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:03.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:03.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:03 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:03.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:03.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:03 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:03.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:03.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:03 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:03.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:04.544 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:04.105+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:04.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:04.105+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:04.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:04.105+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:04.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:04 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:04.105+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:04.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:04.326+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:04.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:04.326+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:04.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:04.326+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:04.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:04.326+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:04.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:04.326+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:04.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:04.468+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:04.468+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:04.468+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:04.468+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:04.419+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:04.419+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:04.419+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:04.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:04.419+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:04.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:04 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:04.419+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:05.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:05.092+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:05.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:05.092+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:05.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:05.092+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:05.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:05 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:05.092+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:05.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:05 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:05.369+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:05.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:05 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:05.369+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:05.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:05 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:05.369+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:05.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:05 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:05.369+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:05.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:05 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:05.369+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:05.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:05.424+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:05.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:05.425+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:05.425+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:05.425+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:05.425+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.531485+0000 front 2024-05-26T12:04:43.531471+0000 (oldest deadline 2024-05-26T12:05:05.231597+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:05.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:05.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:05.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:05.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:05.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:05.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:05 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:05.448+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:06.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:06.062+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:06.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:06.062+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:06.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:06.062+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:06.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:06 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:06.062+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:06.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:06.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:06.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:06.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:06.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:06.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:06 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:06.385+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:06.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:06.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:06.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:06.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:06.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:06.470+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.531485+0000 front 2024-05-26T12:04:43.531471+0000 (oldest deadline 2024-05-26T12:05:05.231597+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:06.434+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:06.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:06.434+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:06.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:06.434+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:06.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:06.434+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:06.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:06 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:06.434+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:07.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:07 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:07.104+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:07.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:07 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:07.104+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:07.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:07 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:07.104+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:07.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:07 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:07.104+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:07.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:07.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:07.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:07.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:07.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:07.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:07 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:07.371+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:07.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:07.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.725958+0000 front 2024-05-26T12:04:09.726005+0000 (oldest deadline 2024-05-26T12:04:33.826045+0000) 2024-05-26T12:05:07.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:07.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.527077+0000 front 2024-05-26T12:04:15.527166+0000 (oldest deadline 2024-05-26T12:04:40.227186+0000) 2024-05-26T12:05:07.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:07.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.928511+0000 front 2024-05-26T12:04:24.928458+0000 (oldest deadline 2024-05-26T12:04:49.628325+0000) 2024-05-26T12:05:07.822 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:07.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.829924+0000 front 2024-05-26T12:04:34.829889+0000 (oldest deadline 2024-05-26T12:05:01.830465+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8[54946]: 2024-05-26T12:05:07.489+0000 7fdede4d1700 -1 osd.8 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.531485+0000 front 2024-05-26T12:04:43.531471+0000 (oldest deadline 2024-05-26T12:05:05.231597+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:07.389+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:07.389+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:07.389+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:07.389+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:07.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:07 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:07.389+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:08.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:08 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:08.076+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:08.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:08 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:08.076+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:08.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:08 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:08.076+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:08.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:08 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:08.076+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:08.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:08 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:08.076+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:08.697 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:08.697 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:08.698 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:08.698 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:08.698 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:08.698 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:08.433+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:08.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:08.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:08.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:08.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:08.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:08.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:08 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:08.347+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:08.976 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:08 smithi192 podman[70019]: 2024-05-26 12:05:08.697009112 +0000 UTC m=+5.542853179 container died ca57e492b83fb88415abd402484f38f377601e6e7e9a33e9717536622a64f330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240521) 2024-05-26T12:05:09.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:09 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:09.042+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:09.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:09 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:09.042+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:09.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:09 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:09.042+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:09.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:09 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:09.042+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:09.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:09 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:09.042+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:09.322 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:09 smithi192 podman[70019]: 2024-05-26 12:05:09.165872464 +0000 UTC m=+6.011716531 container cleanup ca57e492b83fb88415abd402484f38f377601e6e7e9a33e9717536622a64f330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-26T12:05:09.322 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:09 smithi192 bash[70019]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8 2024-05-26T12:05:09.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:09.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:09.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:09.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:09.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:09.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:09 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:09.392+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:09.778 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:09 smithi192 podman[70035]: 2024-05-26 12:05:09.458948692 +0000 UTC m=+0.752618381 container remove ca57e492b83fb88415abd402484f38f377601e6e7e9a33e9717536622a64f330 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-26T12:05:09.778 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:09.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:09.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:09.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:09.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:09.779 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:09 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:09.473+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:10.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:09 smithi192 podman[70127]: 2024-05-26 12:05:09.681223348 +0000 UTC m=+0.017132468 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:05:10.072 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:09 smithi192 podman[70127]: 2024-05-26 12:05:09.784616065 +0000 UTC m=+0.120525217 container create 4d7f841b68ae44bef849246852c699f148e2225331209b0ea389c4d442baf4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-26T12:05:10.073 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:10 smithi192 podman[70127]: 2024-05-26 12:05:10.020928395 +0000 UTC m=+0.356837506 container init 4d7f841b68ae44bef849246852c699f148e2225331209b0ea389c4d442baf4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5) 2024-05-26T12:05:10.073 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:10 smithi192 podman[70127]: 2024-05-26 12:05:10.024387135 +0000 UTC m=+0.360296252 container start 4d7f841b68ae44bef849246852c699f148e2225331209b0ea389c4d442baf4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:10.325 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:10 smithi192 podman[70127]: 2024-05-26 12:05:10.0871287 +0000 UTC m=+0.423037815 container attach 4d7f841b68ae44bef849246852c699f148e2225331209b0ea389c4d442baf4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:10.544 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:10 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:10.059+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:10.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:10 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:10.059+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:10.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:10 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:10.059+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:10.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:10 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:10.059+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:10.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:10 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:10.059+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:10.649 INFO:journalctl@ceph.osd.8.smithi192.stdout:May 26 12:05:10 smithi192 podman[70127]: 2024-05-26 12:05:10.325488641 +0000 UTC m=+0.661397764 container died 4d7f841b68ae44bef849246852c699f148e2225331209b0ea389c4d442baf4f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-8-deactivate, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-26T12:05:10.649 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.445+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:10.649 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.445+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:10.650 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.445+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:10.650 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.445+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:10.650 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.446+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:10.650 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:10 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:10.446+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:10.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:10.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:10.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:10.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:10.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:10.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:10 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:10.403+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:10.950 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.8.service' 2024-05-26T12:05:10.987 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:10.988 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-26T12:05:10.988 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-26T12:05:10.988 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.9 2024-05-26T12:05:11.294 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 systemd[1]: Stopping Ceph osd.9 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:11.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.046+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:11.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.046+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:11.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.046+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:11.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.046+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:11.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.046+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:11.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:11.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:11.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:11.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:11.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:11.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:11 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:11.404+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:11.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:11.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:11.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:11.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:11.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:11.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:11 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:11.389+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:11.794 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.314+0000 7f028aefd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:11.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.314+0000 7f028aefd700 -1 osd.9 86 *** Got signal Terminated *** 2024-05-26T12:05:11.795 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:11 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:11.314+0000 7f028aefd700 -1 osd.9 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:05:12.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.008+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:12.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.008+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:12.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.008+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:12.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.008+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:12.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.008+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:12.476 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:12.476 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:12.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:12.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:12.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:12.477 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:12 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:12.402+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:12.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:12.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:12.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:12.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:12.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:12.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:12.387+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:13.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.997+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:13.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.997+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:13.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.997+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:13.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.997+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:13.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:12 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:12.997+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:13.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:13.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:13.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:13.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:13.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:13.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:13 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:13.364+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:13.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:13.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:13.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:13.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:13.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:13.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:13 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:13.359+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:14.294 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:13 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:13.991+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:14.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:13 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:13.991+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:14.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:13 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:13.991+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:14.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:13 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:13.991+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:14.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:13 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:13.991+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:14.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:14.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:14.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:14.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:14.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:14.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:14.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:14 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:14.325+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:14.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:14.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:14.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:14.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:14.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:14.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:14 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:14.385+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:15.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:14 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:14.976+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:51.850569+0000 front 2024-05-26T12:04:51.850525+0000 (oldest deadline 2024-05-26T12:05:14.750663+0000) 2024-05-26T12:05:15.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:15.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:15 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:15.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:15.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:15.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:15.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:15.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:15.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:15.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:15.342+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:59.942826+0000 front 2024-05-26T12:03:59.942725+0000 (oldest deadline 2024-05-26T12:04:25.843025+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:07.544221+0000 front 2024-05-26T12:04:07.544304+0000 (oldest deadline 2024-05-26T12:04:31.644133+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:18.045637+0000 front 2024-05-26T12:04:18.045721+0000 (oldest deadline 2024-05-26T12:04:43.345635+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:27.346926+0000 front 2024-05-26T12:04:27.346965+0000 (oldest deadline 2024-05-26T12:04:48.447082+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:43.049804+0000 front 2024-05-26T12:04:43.049816+0000 (oldest deadline 2024-05-26T12:05:07.149644+0000) 2024-05-26T12:05:16.295 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9[64063]: 2024-05-26T12:05:16.007+0000 7f02874f1700 -1 osd.9 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:51.850569+0000 front 2024-05-26T12:04:51.850525+0000 (oldest deadline 2024-05-26T12:05:14.750663+0000) 2024-05-26T12:05:16.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:16.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:16 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:16.294+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:16.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:16.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:16.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:16.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:16.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:16.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:16 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:16.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:17.045 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:16 smithi156 podman[85923]: 2024-05-26 12:05:16.747270968 +0000 UTC m=+5.553146737 container died 26127fd2f9103e7a1e5c4542669643373facc0cf2ea37197303152ffac8cac12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-26T12:05:17.544 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[85923]: 2024-05-26 12:05:17.172538836 +0000 UTC m=+5.978414648 container cleanup 26127fd2f9103e7a1e5c4542669643373facc0cf2ea37197303152ffac8cac12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5) 2024-05-26T12:05:17.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 bash[85923]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9 2024-05-26T12:05:17.545 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[85938]: 2024-05-26 12:05:17.407167368 +0000 UTC m=+0.652213421 container remove 26127fd2f9103e7a1e5c4542669643373facc0cf2ea37197303152ffac8cac12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:05:17.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:17.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:17.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:17.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:17.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:17.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:17 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:17.308+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:17.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:17.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:17 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:17.300+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:18.044 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[86027]: 2024-05-26 12:05:17.645926445 +0000 UTC m=+0.017002644 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:05:18.045 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[86027]: 2024-05-26 12:05:17.754848147 +0000 UTC m=+0.125924387 container create 4c888e6255f68413811846a0a62b7872c95a7da3db1c5bfe6397fb8876ba7b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-26T12:05:18.045 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[86027]: 2024-05-26 12:05:17.985613056 +0000 UTC m=+0.356689250 container init 4c888e6255f68413811846a0a62b7872c95a7da3db1c5bfe6397fb8876ba7b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, org.label-schema.build-date=20240521, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-26T12:05:18.045 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:17 smithi156 podman[86027]: 2024-05-26 12:05:17.988879971 +0000 UTC m=+0.359956167 container start 4c888e6255f68413811846a0a62b7872c95a7da3db1c5bfe6397fb8876ba7b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-26T12:05:18.320 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:18 smithi156 podman[86027]: 2024-05-26 12:05:18.077213246 +0000 UTC m=+0.448289471 container attach 4c888e6255f68413811846a0a62b7872c95a7da3db1c5bfe6397fb8876ba7b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, CEPH_POINT_RELEASE=) 2024-05-26T12:05:18.320 INFO:journalctl@ceph.osd.9.smithi156.stdout:May 26 12:05:18 smithi156 podman[86027]: 2024-05-26 12:05:18.287522275 +0000 UTC m=+0.658598477 container died 4c888e6255f68413811846a0a62b7872c95a7da3db1c5bfe6397fb8876ba7b7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-9-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521) 2024-05-26T12:05:18.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:18.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:18 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:18.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:18.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:18.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:18.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:18.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:18.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:18.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:18 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:18.328+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:18.908 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.9.service' 2024-05-26T12:05:18.946 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:18.946 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-26T12:05:18.947 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-26T12:05:18.947 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.10 2024-05-26T12:05:19.226 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 systemd[1]: Stopping Ceph osd.10 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:19.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:19.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:19.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:19.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:19.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:19.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:19 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:19.310+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:19.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.267+0000 7f4546351700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.267+0000 7f4546351700 -1 osd.10 86 *** Got signal Terminated *** 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.267+0000 7f4546351700 -1 osd.10 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:19.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:19.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:19.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:19 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:19.360+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:20.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:20.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:20.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:20.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:20.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:20.574 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:20.276+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:20.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:20.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:20 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:20.321+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:21.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:21.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:21.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:21.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:21.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:21.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:21 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:21.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:21.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:21.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:21 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:21.333+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:22.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:22.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:22.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:22.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:22.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:22.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:22.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:22 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:22.308+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:22.750 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:22.750 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:22.750 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:22.750 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:22.751 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:22 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:22.329+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:23.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:23.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:23.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:23.673 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:23.674 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:23.674 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:23.674 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:23 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:23.336+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:23.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:03:57.598513+0000 front 2024-05-26T12:03:57.598537+0000 (oldest deadline 2024-05-26T12:04:23.498324+0000) 2024-05-26T12:05:23.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:08.799177+0000 front 2024-05-26T12:04:08.799315+0000 (oldest deadline 2024-05-26T12:04:33.499320+0000) 2024-05-26T12:05:23.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:17.600404+0000 front 2024-05-26T12:04:17.600252+0000 (oldest deadline 2024-05-26T12:04:41.700298+0000) 2024-05-26T12:05:23.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:21.701037+0000 front 2024-05-26T12:04:21.701065+0000 (oldest deadline 2024-05-26T12:04:48.101615+0000) 2024-05-26T12:05:23.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.403608+0000 front 2024-05-26T12:04:34.403649+0000 (oldest deadline 2024-05-26T12:04:58.503800+0000) 2024-05-26T12:05:23.727 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:41.405072+0000 front 2024-05-26T12:04:41.404958+0000 (oldest deadline 2024-05-26T12:05:05.504715+0000) 2024-05-26T12:05:23.728 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:23 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10[61882]: 2024-05-26T12:05:23.341+0000 7f4542945700 -1 osd.10 86 heartbeat_check: no reply from 172.21.15.156:6822 osd.6 since back 2024-05-26T12:04:48.405873+0000 front 2024-05-26T12:04:48.405931+0000 (oldest deadline 2024-05-26T12:05:14.305614+0000) 2024-05-26T12:05:24.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:24.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:24.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:24.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:24.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:24.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:24.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:24 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:24.372+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:25.044 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:24 smithi186 podman[70636]: 2024-05-26 12:05:24.744058818 +0000 UTC m=+5.594723591 container died 2b96e9b522703e551e4bf0df35989fec79373b7dfd8cf57b49f44cd650d959e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS) 2024-05-26T12:05:25.368 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:25 smithi186 podman[70636]: 2024-05-26 12:05:25.209332013 +0000 UTC m=+6.059996785 container cleanup 2b96e9b522703e551e4bf0df35989fec79373b7dfd8cf57b49f44cd650d959e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-05-26T12:05:25.369 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:25 smithi186 bash[70636]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10 2024-05-26T12:05:25.726 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:25 smithi186 podman[70650]: 2024-05-26 12:05:25.368897333 +0000 UTC m=+0.620128998 container remove 2b96e9b522703e551e4bf0df35989fec79373b7dfd8cf57b49f44cd650d959e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10, org.label-schema.build-date=20240521, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:25.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:25.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:25.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:25.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:25.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:25.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:25.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:25 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:25.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:26.089 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:25 smithi186 podman[70740]: 2024-05-26 12:05:25.786706009 +0000 UTC m=+0.105939260 container create 13e53045da09c5cce858577736d7efc5c9b45f98b008d69e9b09673c5d9d51cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:26.089 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:25 smithi186 podman[70740]: 2024-05-26 12:05:25.697903418 +0000 UTC m=+0.017136675 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:05:26.089 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:26 smithi186 podman[70740]: 2024-05-26 12:05:26.005802309 +0000 UTC m=+0.325035531 container init 13e53045da09c5cce858577736d7efc5c9b45f98b008d69e9b09673c5d9d51cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10-deactivate, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-26T12:05:26.089 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:26 smithi186 podman[70740]: 2024-05-26 12:05:26.009623802 +0000 UTC m=+0.328857035 container start 13e53045da09c5cce858577736d7efc5c9b45f98b008d69e9b09673c5d9d51cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-26T12:05:26.089 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:26 smithi186 podman[70740]: 2024-05-26 12:05:26.088857899 +0000 UTC m=+0.408091128 container attach 13e53045da09c5cce858577736d7efc5c9b45f98b008d69e9b09673c5d9d51cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:26.476 INFO:journalctl@ceph.osd.10.smithi186.stdout:May 26 12:05:26 smithi186 podman[70740]: 2024-05-26 12:05:26.30634065 +0000 UTC m=+0.625573886 container died 13e53045da09c5cce858577736d7efc5c9b45f98b008d69e9b09673c5d9d51cd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:26.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:26.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:26.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.364+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:26.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.365+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:26.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.365+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:26.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.365+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:26.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:26.365+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:26.888 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.10.service' 2024-05-26T12:05:26.925 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:26.925 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-26T12:05:26.925 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-26T12:05:26.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.11 2024-05-26T12:05:27.243 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:26 smithi192 systemd[1]: Stopping Ceph osd.11 for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:27.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.242+0000 7fd687efc700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:27.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.242+0000 7fd687efc700 -1 osd.11 86 *** Got signal Terminated *** 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.242+0000 7fd687efc700 -1 osd.11 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:27.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:27 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:27.315+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:28.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:28.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:28.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:28.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:28.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:28.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:28.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:28 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:28.286+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:29.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:29.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:29.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:29.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:29.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:29.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:29.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:29 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:29.279+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:30.822 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:30.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:30.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:30.823 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:30.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:30.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:30.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:30.824 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:30 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:30.325+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-05-26T12:05:04.335897+0000 front 2024-05-26T12:05:04.335944+0000 (oldest deadline 2024-05-26T12:05:29.635958+0000) 2024-05-26T12:05:31.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6806 osd.0 since back 2024-05-26T12:04:00.425717+0000 front 2024-05-26T12:04:00.425741+0000 (oldest deadline 2024-05-26T12:04:24.525578+0000) 2024-05-26T12:05:31.572 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6804 osd.1 since back 2024-05-26T12:04:09.726991+0000 front 2024-05-26T12:04:09.726887+0000 (oldest deadline 2024-05-26T12:04:32.627000+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6804 osd.2 since back 2024-05-26T12:04:15.929291+0000 front 2024-05-26T12:04:15.929366+0000 (oldest deadline 2024-05-26T12:04:41.129885+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.156:6814 osd.3 since back 2024-05-26T12:04:24.030900+0000 front 2024-05-26T12:04:24.030934+0000 (oldest deadline 2024-05-26T12:04:49.330716+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6812 osd.4 since back 2024-05-26T12:04:34.631758+0000 front 2024-05-26T12:04:34.631743+0000 (oldest deadline 2024-05-26T12:04:58.131723+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6812 osd.5 since back 2024-05-26T12:04:42.832607+0000 front 2024-05-26T12:04:42.832587+0000 (oldest deadline 2024-05-26T12:05:07.532614+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.186:6820 osd.7 since back 2024-05-26T12:04:59.834645+0000 front 2024-05-26T12:04:59.834591+0000 (oldest deadline 2024-05-26T12:05:22.134436+0000) 2024-05-26T12:05:31.573 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:31 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11[61816]: 2024-05-26T12:05:31.306+0000 7fd6804e8700 -1 osd.11 86 heartbeat_check: no reply from 172.21.15.192:6820 osd.8 since back 2024-05-26T12:05:04.335897+0000 front 2024-05-26T12:05:04.335944+0000 (oldest deadline 2024-05-26T12:05:29.635958+0000) 2024-05-26T12:05:33.043 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:32 smithi192 podman[70259]: 2024-05-26 12:05:32.741822373 +0000 UTC m=+5.623669655 container died a54259b3109f2ae34822a8565e487377c414fe20b06d65ff166bfbf27843427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True) 2024-05-26T12:05:33.322 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70259]: 2024-05-26 12:05:33.134245168 +0000 UTC m=+6.016092440 container cleanup a54259b3109f2ae34822a8565e487377c414fe20b06d65ff166bfbf27843427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:33.322 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 bash[70259]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11 2024-05-26T12:05:33.669 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70277]: 2024-05-26 12:05:33.360395412 +0000 UTC m=+0.609588383 container remove a54259b3109f2ae34822a8565e487377c414fe20b06d65ff166bfbf27843427c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-26T12:05:33.997 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70366]: 2024-05-26 12:05:33.669547793 +0000 UTC m=+0.101766250 container create f771b9b624875e7ad6fafa932fb0f981ac254950b540108be18fc62c50eee765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-26T12:05:33.997 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70366]: 2024-05-26 12:05:33.585562481 +0000 UTC m=+0.017780896 image pull 939e725847fbf78c162f64b8527a03b31d76f3728e38f2dd006f78157d61d409 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b 2024-05-26T12:05:33.997 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70366]: 2024-05-26 12:05:33.913831275 +0000 UTC m=+0.346049689 container init f771b9b624875e7ad6fafa932fb0f981ac254950b540108be18fc62c50eee765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-05-26T12:05:33.998 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70366]: 2024-05-26 12:05:33.916878769 +0000 UTC m=+0.349097192 container start f771b9b624875e7ad6fafa932fb0f981ac254950b540108be18fc62c50eee765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-26T12:05:34.322 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:33 smithi192 podman[70366]: 2024-05-26 12:05:33.996945822 +0000 UTC m=+0.429164255 container attach f771b9b624875e7ad6fafa932fb0f981ac254950b540108be18fc62c50eee765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-05-26T12:05:34.323 INFO:journalctl@ceph.osd.11.smithi192.stdout:May 26 12:05:34 smithi192 podman[70366]: 2024-05-26 12:05:34.206046325 +0000 UTC m=+0.638264741 container died f771b9b624875e7ad6fafa932fb0f981ac254950b540108be18fc62c50eee765 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:34.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@osd.11.service' 2024-05-26T12:05:34.874 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:34.874 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-26T12:05:34.874 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-26T12:05:34.874 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.a 2024-05-26T12:05:35.167 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:34 smithi156 systemd[1]: Stopping Ceph mds.a for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:35.544 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a[71023]: 2024-05-26T12:05:35.165+0000 7f90b895e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:35.545 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:35 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a[71023]: 2024-05-26T12:05:35.165+0000 7f90b895e700 -1 mds.a *** got signal Terminated *** 2024-05-26T12:05:39.175 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:38 smithi156 podman[86143]: 2024-05-26 12:05:38.895582898 +0000 UTC m=+3.823637968 container died e0f4feea93d80498fec38f2aca256215d5daec2b9d483786210e79bf87e9b718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-05-26T12:05:39.544 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:39 smithi156 podman[86143]: 2024-05-26 12:05:39.365448769 +0000 UTC m=+4.293503840 container cleanup e0f4feea93d80498fec38f2aca256215d5daec2b9d483786210e79bf87e9b718 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-26T12:05:39.545 INFO:journalctl@ceph.mds.a.smithi156.stdout:May 26 12:05:39 smithi156 bash[86143]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-a 2024-05-26T12:05:39.806 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.a.service' 2024-05-26T12:05:39.841 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:39.841 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-26T12:05:39.841 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-26T12:05:39.842 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.d 2024-05-26T12:05:40.294 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:39 smithi156 systemd[1]: Stopping Ceph mds.d for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:40.295 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d[69653]: 2024-05-26T12:05:40.143+0000 7f5759027700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:40.295 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:40 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d[69653]: 2024-05-26T12:05:40.143+0000 7f5759027700 -1 mds.d *** got signal Terminated *** 2024-05-26T12:05:44.544 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:44 smithi156 podman[86263]: 2024-05-26 12:05:44.150262643 +0000 UTC m=+4.085128838 container died 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:05:44.893 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:44 smithi156 podman[86263]: 2024-05-26 12:05:44.55224882 +0000 UTC m=+4.487115014 container cleanup 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-05-26T12:05:44.893 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:44 smithi156 bash[86263]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d 2024-05-26T12:05:44.893 INFO:journalctl@ceph.mds.d.smithi156.stdout:May 26 12:05:44 smithi156 podman[86277]: 2024-05-26 12:05:44.770399277 +0000 UTC m=+0.614622409 container remove 0f0a5733d02abc2e24f46530e7c2dcaec56f6ce70f78a88cdd1f57bbc1ee9b45 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.build-date=20240521, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-05-26T12:05:44.910 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.d.service' 2024-05-26T12:05:44.943 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:44.943 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-26T12:05:44.943 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-26T12:05:44.944 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.g 2024-05-26T12:05:45.273 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:45 smithi156 systemd[1]: Stopping Ceph mds.g for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:45.544 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: 2024-05-26T12:05:45.272+0000 7f3f6bbc4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:45.545 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:45 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g[70580]: 2024-05-26T12:05:45.272+0000 7f3f6bbc4700 -1 mds.g *** got signal Terminated *** 2024-05-26T12:05:47.207 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:46 smithi156 podman[86383]: 2024-05-26 12:05:46.893110312 +0000 UTC m=+1.730190732 container died 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:47.544 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:47 smithi156 podman[86383]: 2024-05-26 12:05:47.382064796 +0000 UTC m=+2.219145215 container cleanup 8d8891dfe1540b24e7a9a5612bc8a94036c6eed934a9360bf81ed5b8b4a6f961 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, GIT_CLEAN=True) 2024-05-26T12:05:47.545 INFO:journalctl@ceph.mds.g.smithi156.stdout:May 26 12:05:47 smithi156 bash[86383]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-g 2024-05-26T12:05:47.778 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.g.service' 2024-05-26T12:05:47.810 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:47.810 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-26T12:05:47.810 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-26T12:05:47.810 DEBUG:teuthology.orchestra.run.smithi156:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.j 2024-05-26T12:05:48.294 INFO:journalctl@ceph.mds.j.smithi156.stdout:May 26 12:05:48 smithi156 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-j[69315]: 2024-05-26T12:05:48.135+0000 7f6e5c1d8700 -1 mds.j *** got signal Terminated *** 2024-05-26T12:05:50.044 INFO:journalctl@ceph.mds.j.smithi156.stdout:May 26 12:05:49 smithi156 podman[86503]: 2024-05-26 12:05:49.722251623 +0000 UTC m=+1.689163892 container died 134f94b5d2247d5125bac48d19a81db77f8b1eec6286f635d44096b40dd53117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-j, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-26T12:05:50.470 INFO:journalctl@ceph.mds.j.smithi156.stdout:May 26 12:05:50 smithi156 podman[86503]: 2024-05-26 12:05:50.227658578 +0000 UTC m=+2.194570795 container cleanup 134f94b5d2247d5125bac48d19a81db77f8b1eec6286f635d44096b40dd53117 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-26T12:05:50.470 INFO:journalctl@ceph.mds.j.smithi156.stdout:May 26 12:05:50 smithi156 bash[86503]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-j 2024-05-26T12:05:50.606 DEBUG:teuthology.orchestra.run.smithi156:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.j.service' 2024-05-26T12:05:50.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:50.638 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-26T12:05:50.639 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-26T12:05:50.639 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.b 2024-05-26T12:05:50.944 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:50 smithi186 systemd[1]: Stopping Ceph mds.b for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:51.226 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: 2024-05-26T12:05:50.942+0000 7fb50c0b9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:51.226 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:50 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b[67534]: 2024-05-26T12:05:50.942+0000 7fb50c0b9700 -1 mds.b *** got signal Terminated *** 2024-05-26T12:05:55.193 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:54 smithi186 podman[70857]: 2024-05-26 12:05:54.910079631 +0000 UTC m=+4.078447427 container died 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-05-26T12:05:55.476 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:55 smithi186 podman[70857]: 2024-05-26 12:05:55.292881072 +0000 UTC m=+4.461248865 container cleanup 30a3efad7f4e1b803ccac49a8a7049bec00365e67c811f4e7daeb1979c493178 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:05:55.476 INFO:journalctl@ceph.mds.b.smithi186.stdout:May 26 12:05:55 smithi186 bash[70857]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-b 2024-05-26T12:05:55.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.b.service' 2024-05-26T12:05:55.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:55.792 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-26T12:05:55.792 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-26T12:05:55.792 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.e 2024-05-26T12:05:56.226 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:55 smithi186 systemd[1]: Stopping Ceph mds.e for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:05:56.226 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: 2024-05-26T12:05:56.096+0000 7f782c755700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:05:56.227 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:56 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e[66910]: 2024-05-26T12:05:56.096+0000 7f782c755700 -1 mds.e *** got signal Terminated *** 2024-05-26T12:05:59.226 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:58 smithi186 podman[70978]: 2024-05-26 12:05:58.905015108 +0000 UTC m=+2.897354492 container died 71176291a1a048a2d162141caa0f2675cccb87b3d4ce4ffac351888cb940a9d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.build-date=20240521, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-26T12:05:59.540 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:59 smithi186 podman[70978]: 2024-05-26 12:05:59.338596852 +0000 UTC m=+3.330936235 container cleanup 71176291a1a048a2d162141caa0f2675cccb87b3d4ce4ffac351888cb940a9d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-26T12:05:59.540 INFO:journalctl@ceph.mds.e.smithi186.stdout:May 26 12:05:59 smithi186 bash[70978]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-e 2024-05-26T12:05:59.765 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.e.service' 2024-05-26T12:05:59.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:05:59.799 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-26T12:05:59.799 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-26T12:05:59.799 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.h 2024-05-26T12:06:00.118 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:05:59 smithi186 systemd[1]: Stopping Ceph mds.h for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:00.476 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: 2024-05-26T12:06:00.117+0000 7f2637d3a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:00.476 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:00 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h[67219]: 2024-05-26T12:06:00.117+0000 7f2637d3a700 -1 mds.h *** got signal Terminated *** 2024-05-26T12:06:03.226 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:02 smithi186 podman[71112]: 2024-05-26 12:06:02.895141396 +0000 UTC m=+2.882470433 container died b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-26T12:06:03.535 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 podman[71112]: 2024-05-26 12:06:03.342810441 +0000 UTC m=+3.330139435 container cleanup b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240521, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-26T12:06:03.536 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 bash[71112]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h 2024-05-26T12:06:03.775 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.h.service' 2024-05-26T12:06:03.795 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 podman[71124]: 2024-05-26 12:06:03.535588204 +0000 UTC m=+0.630039456 container remove b2fbf4ed172301e6aaddf8f6f5ab040096bcd3bd8285adc496b98ab34a54a7ed (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-h, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.vendor=CentOS) 2024-05-26T12:06:03.795 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.h.service: Deactivated successfully. 2024-05-26T12:06:03.795 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 systemd[1]: Stopped Ceph mds.h for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:06:03.796 INFO:journalctl@ceph.mds.h.smithi186.stdout:May 26 12:06:03 smithi186 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.h.service: Consumed 1.175s CPU time. 2024-05-26T12:06:03.807 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:03.807 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-26T12:06:03.807 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-26T12:06:03.807 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.k 2024-05-26T12:06:04.226 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:03 smithi186 systemd[1]: Stopping Ceph mds.k for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:04.726 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: 2024-05-26T12:06:04.262+0000 7f2e5457d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:04.727 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:04 smithi186 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k[66596]: 2024-05-26T12:06:04.262+0000 7f2e5457d700 -1 mds.k *** got signal Terminated *** 2024-05-26T12:06:07.226 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:06 smithi186 podman[71234]: 2024-05-26 12:06:06.897255856 +0000 UTC m=+2.868204963 container died 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240521, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-26T12:06:07.531 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 podman[71234]: 2024-05-26 12:06:07.338187599 +0000 UTC m=+3.309136697 container cleanup 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.build-date=20240521, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-05-26T12:06:07.531 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 bash[71234]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k 2024-05-26T12:06:07.762 DEBUG:teuthology.orchestra.run.smithi186:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.k.service' 2024-05-26T12:06:07.782 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 podman[71246]: 2024-05-26 12:06:07.531026886 +0000 UTC m=+0.632136158 container remove 84620862ab393b16e22e9ff468ad20ba346d6e8841204ce38c37c1b1169895ea (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-lusov-quiesce-9a4c585, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS) 2024-05-26T12:06:07.782 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.k.service: Deactivated successfully. 2024-05-26T12:06:07.783 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 systemd[1]: Stopped Ceph mds.k for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:06:07.783 INFO:journalctl@ceph.mds.k.smithi186.stdout:May 26 12:06:07 smithi186 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.k.service: Consumed 1.159s CPU time. 2024-05-26T12:06:07.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:07.793 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-26T12:06:07.794 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-26T12:06:07.794 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.c 2024-05-26T12:06:08.072 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:07 smithi192 systemd[1]: Stopping Ceph mds.c for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:08.341 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c[66807]: 2024-05-26T12:06:08.090+0000 7f78e1430700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:08.342 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:08 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c[66807]: 2024-05-26T12:06:08.090+0000 7f78e1430700 -1 mds.c *** got signal Terminated *** 2024-05-26T12:06:11.242 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:10 smithi192 podman[70473]: 2024-05-26 12:06:10.898395458 +0000 UTC m=+2.903426461 container died b43d74acfb02452ec5c47c9902f51a7870bb2d7c8e36dc28a5102775c64abb85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-lusov-quiesce-9a4c585) 2024-05-26T12:06:11.516 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 podman[70473]: 2024-05-26 12:06:11.332226005 +0000 UTC m=+3.337257002 container cleanup b43d74acfb02452ec5c47c9902f51a7870bb2d7c8e36dc28a5102775c64abb85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.build-date=20240521, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, RELEASE=wip-lusov-quiesce-9a4c585, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-26T12:06:11.517 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 bash[70473]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c 2024-05-26T12:06:11.763 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.c.service' 2024-05-26T12:06:11.784 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 podman[70499]: 2024-05-26 12:06:11.516576452 +0000 UTC m=+0.612835064 container remove b43d74acfb02452ec5c47c9902f51a7870bb2d7c8e36dc28a5102775c64abb85 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-c, ceph=True, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-26T12:06:11.784 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.c.service: Deactivated successfully. 2024-05-26T12:06:11.784 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 systemd[1]: Stopped Ceph mds.c for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:06:11.785 INFO:journalctl@ceph.mds.c.smithi192.stdout:May 26 12:06:11 smithi192 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.c.service: Consumed 1.266s CPU time. 2024-05-26T12:06:11.798 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:11.798 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-26T12:06:11.798 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-26T12:06:11.799 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.f 2024-05-26T12:06:12.072 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:11 smithi192 systemd[1]: Stopping Ceph mds.f for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:12.572 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: 2024-05-26T12:06:12.127+0000 7f0b4ec38700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:12.572 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:12 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f[66503]: 2024-05-26T12:06:12.127+0000 7f0b4ec38700 -1 mds.f *** got signal Terminated *** 2024-05-26T12:06:15.178 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:14 smithi192 podman[70605]: 2024-05-26 12:06:14.903502967 +0000 UTC m=+2.885973609 container died 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-26T12:06:15.495 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:15 smithi192 podman[70605]: 2024-05-26 12:06:15.269332258 +0000 UTC m=+3.251802882 container cleanup 2e245d9e4126d647f0e091a7ba1540158edc08d54087d67ad5034eebc9155d90 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f, org.label-schema.build-date=20240521, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-lusov-quiesce-9a4c585, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-26T12:06:15.495 INFO:journalctl@ceph.mds.f.smithi192.stdout:May 26 12:06:15 smithi192 bash[70605]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-f 2024-05-26T12:06:15.624 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.f.service' 2024-05-26T12:06:15.655 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:15.656 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-26T12:06:15.656 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-26T12:06:15.656 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.i 2024-05-26T12:06:15.822 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:15 smithi192 systemd[1]: Stopping Ceph mds.i for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:16.322 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: 2024-05-26T12:06:15.963+0000 7f3c4adc7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:16.322 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:15 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i[67111]: 2024-05-26T12:06:15.963+0000 7f3c4adc7700 -1 mds.i *** got signal Terminated *** 2024-05-26T12:06:19.072 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:18 smithi192 podman[70725]: 2024-05-26 12:06:18.903547171 +0000 UTC m=+3.035871262 container died 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-lusov-quiesce-9a4c585, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-26T12:06:19.473 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 podman[70725]: 2024-05-26 12:06:19.239191 +0000 UTC m=+3.371515083 container cleanup 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, org.label-schema.build-date=20240521, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-lusov-quiesce-9a4c585, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:06:19.473 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 bash[70725]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i 2024-05-26T12:06:19.715 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.i.service' 2024-05-26T12:06:19.735 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 podman[70738]: 2024-05-26 12:06:19.473564497 +0000 UTC m=+0.570599960 container remove 0fca86089d87d1cad5a6987e77419aba4743949644417a26260998acfcdadf09 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-i, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240521, CEPH_POINT_RELEASE=) 2024-05-26T12:06:19.735 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.i.service: Deactivated successfully. 2024-05-26T12:06:19.735 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 systemd[1]: Stopped Ceph mds.i for 38c1aff8-1b56-11ef-bc9b-c7b262605968. 2024-05-26T12:06:19.735 INFO:journalctl@ceph.mds.i.smithi192.stdout:May 26 12:06:19 smithi192 systemd[1]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.i.service: Consumed 1.154s CPU time. 2024-05-26T12:06:19.746 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:19.746 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-26T12:06:19.746 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-26T12:06:19.746 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.l 2024-05-26T12:06:20.067 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:19 smithi192 systemd[1]: Stopping Ceph mds.l for 38c1aff8-1b56-11ef-bc9b-c7b262605968... 2024-05-26T12:06:20.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: 2024-05-26T12:06:20.067+0000 7f3013130700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-26T12:06:20.322 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:20 smithi192 ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l[66200]: 2024-05-26T12:06:20.067+0000 7f3013130700 -1 mds.l *** got signal Terminated *** 2024-05-26T12:06:23.251 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:22 smithi192 podman[70845]: 2024-05-26 12:06:22.906827885 +0000 UTC m=+2.943924428 container died ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240521, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-lusov-quiesce-9a4c585, org.label-schema.license=GPLv2, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093) 2024-05-26T12:06:23.526 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:23 smithi192 podman[70845]: 2024-05-26 12:06:23.325525562 +0000 UTC m=+3.362622105 container cleanup ba3e175c640b05f358abe3c57735e65e7fdaacf656bf70a4da85a62069d25a88 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:48df7bf1b210904bbb9092c896894eb4d87ac0aa7dce350fc07d13a5eb2c756b, name=ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l, RELEASE=wip-lusov-quiesce-9a4c585, GIT_COMMIT=e5230d3b004eecd2f42a9b578ebb9642654e8093, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240521, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-26T12:06:23.527 INFO:journalctl@ceph.mds.l.smithi192.stdout:May 26 12:06:23 smithi192 bash[70845]: ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968-mds-l 2024-05-26T12:06:23.744 DEBUG:teuthology.orchestra.run.smithi192:> sudo pkill -f 'journalctl -f -n 0 -u ceph-38c1aff8-1b56-11ef-bc9b-c7b262605968@mds.l.service' 2024-05-26T12:06:23.778 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-26T12:06:23.778 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-26T12:06:23.779 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-26T12:06:23.971 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:26.437 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-26T12:06:26.627 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:29.080 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force --keep-logs 2024-05-26T12:06:29.270 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:31.610 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:06:31.643 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:06:31.675 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-26T12:06:31.700 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-26T12:06:31.702 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi156/crash 2024-05-26T12:06:31.703 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash -- . 2024-05-26T12:06:31.743 INFO:teuthology.orchestra.run.smithi156.stderr:tar: /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-26T12:06:31.743 INFO:teuthology.orchestra.run.smithi156.stderr:tar: Error is not recoverable: exiting now 2024-05-26T12:06:31.745 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi186/crash 2024-05-26T12:06:31.746 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash -- . 2024-05-26T12:06:31.787 INFO:teuthology.orchestra.run.smithi186.stderr:tar: /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-26T12:06:31.787 INFO:teuthology.orchestra.run.smithi186.stderr:tar: Error is not recoverable: exiting now 2024-05-26T12:06:31.789 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi192/crash 2024-05-26T12:06:31.790 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash -- . 2024-05-26T12:06:31.819 INFO:teuthology.orchestra.run.smithi192.stderr:tar: /var/lib/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/crash: Cannot open: No such file or directory 2024-05-26T12:06:31.820 INFO:teuthology.orchestra.run.smithi192.stderr:tar: Error is not recoverable: exiting now 2024-05-26T12:06:31.821 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-26T12:06:31.821 DEBUG:teuthology.orchestra.run.smithi156:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | head -n 1 2024-05-26T12:06:31.857 INFO:tasks.cephadm:Compressing logs... 2024-05-26T12:06:31.857 DEBUG:teuthology.orchestra.run.smithi156:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-26T12:06:31.900 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-26T12:06:31.902 DEBUG:teuthology.orchestra.run.smithi192:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-26T12:06:31.925 INFO:teuthology.orchestra.run.smithi156.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-26T12:06:31.925 INFO:teuthology.orchestra.run.smithi156.stderr:: No such file or directory 2024-05-26T12:06:31.925 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log 2024-05-26T12:06:31.933 INFO:teuthology.orchestra.run.smithi192.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-26T12:06:31.933 INFO:teuthology.orchestra.run.smithi192.stderr:: No such file or directory 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.2.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.f.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.5.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.f.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.c.log 2024-05-26T12:06:31.934 INFO:teuthology.orchestra.run.smithi192.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.c.log 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr: 86.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.z.log 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.i.log 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.i.log: 90.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-26T12:06:31.935 INFO:teuthology.orchestra.run.smithi192.stderr: 92.1% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.z.log.gz 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.g.log 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.j.log 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.3.log 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.g.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.0.log 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.3.log: gzip -5 86.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-26T12:06:31.942 INFO:teuthology.orchestra.run.smithi156.stderr: --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.x.log 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.a.log 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.6.log 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr: 90.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.a.log: 94.8% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.j.log.gz 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.a.log 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log: 83.0% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-26T12:06:31.943 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-26T12:06:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.a.log: 93.6% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-26T12:06:31.944 INFO:teuthology.orchestra.run.smithi156.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.d.log 2024-05-26T12:06:31.946 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.11.log 2024-05-26T12:06:31.946 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.8.log 2024-05-26T12:06:31.947 INFO:teuthology.orchestra.run.smithi192.stderr: 82.7% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-26T12:06:31.947 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.11.log: -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-26T12:06:31.949 INFO:teuthology.orchestra.run.smithi186.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-26T12:06:31.949 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-26T12:06:31.949 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log 2024-05-26T12:06:31.949 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log 2024-05-26T12:06:31.949 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.7.log 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.b.log 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.7.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.4.log 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.1.log 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.b.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- 86.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.log.gz 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.y.log 2024-05-26T12:06:31.950 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.h.log 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.y.log: 90.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.audit.log.gz 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.10.log 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.h.log: --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.e.log 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log: 82.8% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph.cephadm.log.gz 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.e.log: 91.9% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.y.log.gz 2024-05-26T12:06:31.951 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.b.log 2024-05-26T12:06:31.952 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.8.log: 93.2% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.f.log.gz 2024-05-26T12:06:31.953 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.9.log 2024-05-26T12:06:31.954 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.d.log: 94.8% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.d.log.gz 2024-05-26T12:06:31.954 INFO:teuthology.orchestra.run.smithi156.stderr: 91.0% 93.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.g.log.gz 2024-05-26T12:06:31.954 INFO:teuthology.orchestra.run.smithi156.stderr: -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-26T12:06:31.955 INFO:teuthology.orchestra.run.smithi192.stderr: 92.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.i.log.gz 2024-05-26T12:06:31.955 INFO:teuthology.orchestra.run.smithi192.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.l.log 2024-05-26T12:06:31.957 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log: 92.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.h.log.gz 2024-05-26T12:06:31.957 INFO:teuthology.orchestra.run.smithi156.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.9.log: 93.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.a.log.gz 2024-05-26T12:06:31.958 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.k.log 2024-05-26T12:06:31.962 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.b.log: 91.1% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-26T12:06:31.962 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.e.log.gz 2024-05-26T12:06:31.963 INFO:teuthology.orchestra.run.smithi192.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log: /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.l.log: 92.9% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.c.log.gz 2024-05-26T12:06:31.967 INFO:teuthology.orchestra.run.smithi192.stderr: 91.0% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-volume.log.gz 2024-05-26T12:06:31.979 INFO:teuthology.orchestra.run.smithi192.stderr: 93.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.l.log.gz 2024-05-26T12:06:31.989 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.k.log: 91.9% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.k.log.gz 2024-05-26T12:06:31.992 INFO:teuthology.orchestra.run.smithi186.stderr: 92.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mds.b.log.gz 2024-05-26T12:06:32.085 INFO:teuthology.orchestra.run.smithi192.stderr: 91.9% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.c.log.gz 2024-05-26T12:06:32.118 INFO:teuthology.orchestra.run.smithi156.stderr: 89.6% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mgr.x.log.gz 2024-05-26T12:06:32.171 INFO:teuthology.orchestra.run.smithi186.stderr: 91.9% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.b.log.gz 2024-05-26T12:06:32.188 INFO:teuthology.orchestra.run.smithi156.stderr: 92.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.9.log.gz 2024-05-26T12:06:32.248 INFO:teuthology.orchestra.run.smithi192.stderr: 92.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.8.log.gz 2024-05-26T12:06:32.255 INFO:teuthology.orchestra.run.smithi192.stderr: 92.2% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.2.log.gz 2024-05-26T12:06:32.258 INFO:teuthology.orchestra.run.smithi186.stderr: 92.5% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.10.log.gz 2024-05-26T12:06:32.272 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.7.log.gz 2024-05-26T12:06:32.285 INFO:teuthology.orchestra.run.smithi156.stderr: 92.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.6.log.gz 2024-05-26T12:06:32.297 INFO:teuthology.orchestra.run.smithi156.stderr: 92.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.3.log.gz 2024-05-26T12:06:32.311 INFO:teuthology.orchestra.run.smithi186.stderr: 92.2% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.1.log.gz 2024-05-26T12:06:32.430 INFO:teuthology.orchestra.run.smithi156.stderr: 90.7% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-mon.a.log.gz 2024-05-26T12:06:32.604 INFO:teuthology.orchestra.run.smithi192.stderr: 93.6% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.11.log.gz 2024-05-26T12:06:32.627 INFO:teuthology.orchestra.run.smithi156.stderr: 93.4% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.0.log.gz 2024-05-26T12:06:32.630 INFO:teuthology.orchestra.run.smithi156.stderr: 2024-05-26T12:06:32.630 INFO:teuthology.orchestra.run.smithi156.stderr:real 0m0.716s 2024-05-26T12:06:32.630 INFO:teuthology.orchestra.run.smithi156.stderr:user 0m2.365s 2024-05-26T12:06:32.630 INFO:teuthology.orchestra.run.smithi156.stderr:sys 0m0.103s 2024-05-26T12:06:32.660 INFO:teuthology.orchestra.run.smithi192.stderr: 93.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.5.log.gz 2024-05-26T12:06:32.662 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-05-26T12:06:32.663 INFO:teuthology.orchestra.run.smithi192.stderr:real 0m0.746s 2024-05-26T12:06:32.663 INFO:teuthology.orchestra.run.smithi192.stderr:user 0m2.249s 2024-05-26T12:06:32.663 INFO:teuthology.orchestra.run.smithi192.stderr:sys 0m0.100s 2024-05-26T12:06:32.675 INFO:teuthology.orchestra.run.smithi186.stderr: 93.3% -- replaced with /var/log/ceph/38c1aff8-1b56-11ef-bc9b-c7b262605968/ceph-osd.4.log.gz 2024-05-26T12:06:32.676 INFO:teuthology.orchestra.run.smithi186.stderr: 2024-05-26T12:06:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m0.760s 2024-05-26T12:06:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m2.118s 2024-05-26T12:06:32.677 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m0.093s 2024-05-26T12:06:32.677 INFO:tasks.cephadm:Archiving logs... 2024-05-26T12:06:32.677 DEBUG:teuthology.misc:Transferring archived files from smithi156:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi156/log 2024-05-26T12:06:32.679 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-26T12:06:33.041 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi186/log 2024-05-26T12:06:33.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-26T12:06:33.363 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi192/log 2024-05-26T12:06:33.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-26T12:06:33.703 INFO:tasks.cephadm:Removing cluster... 2024-05-26T12:06:33.703 DEBUG:teuthology.orchestra.run.smithi156:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force 2024-05-26T12:06:33.886 INFO:teuthology.orchestra.run.smithi156.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:34.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force 2024-05-26T12:06:34.336 INFO:teuthology.orchestra.run.smithi186.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:34.602 DEBUG:teuthology.orchestra.run.smithi192:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 38c1aff8-1b56-11ef-bc9b-c7b262605968 --force 2024-05-26T12:06:34.790 INFO:teuthology.orchestra.run.smithi192.stdout:Deleting cluster with fsid: 38c1aff8-1b56-11ef-bc9b-c7b262605968 2024-05-26T12:06:35.096 INFO:tasks.cephadm:Removing cephadm ... 2024-05-26T12:06:35.096 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-26T12:06:35.112 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-26T12:06:35.128 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-26T12:06:35.143 INFO:tasks.cephadm:Teardown complete 2024-05-26T12:06:35.144 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-26T12:06:35.154 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-26T12:06:35.154 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-26T12:06:35.156 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-26T12:06:35.173 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-26T12:06:35.275 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-26T12:06:35.286 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-26T12:06:35.287 DEBUG:teuthology.orchestra.run.smithi156:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T12:06:35.289 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T12:06:35.291 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-26T12:06:35.301 INFO:teuthology.orchestra.run.smithi156.stderr:bash: line 1: ntpq: command not found 2024-05-26T12:06:35.303 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2024-05-26T12:06:35.305 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:=============================================================================== 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +57us[ +67us] +/- 57ms 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 8 -245us[ -245us] +/- 80ms 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +247us[ +257us] +/- 94ms 2024-05-26T12:06:35.306 INFO:teuthology.orchestra.run.smithi156.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T12:06:35.307 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T12:06:35.307 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2024-05-26T12:06:35.308 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv01.front.sepia.ceph.com 3 6 377 9 +63us[ +65us] +/- 57ms 2024-05-26T12:06:35.308 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 6 -271us[ -271us] +/- 80ms 2024-05-26T12:06:35.308 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +264us[ +264us] +/- 94ms 2024-05-26T12:06:35.308 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T12:06:35.309 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-26T12:06:35.309 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-05-26T12:06:35.309 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv01.front.sepia.ceph.com 3 6 377 8 +48us[ +38us] +/- 57ms 2024-05-26T12:06:35.310 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 11 -272us[ -282us] +/- 79ms 2024-05-26T12:06:35.310 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 12 +247us[ +236us] +/- 94ms 2024-05-26T12:06:35.310 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-05-26T12:06:35.311 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-26T12:06:35.320 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-26T12:06:35.320 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-26T12:06:35.335 DEBUG:teuthology.orchestra.run.smithi156:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-26T12:06:35.416 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-26T12:06:35.479 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-26T12:06:35.542 DEBUG:teuthology.orchestra.run.smithi156:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T12:06:35.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:35.572 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T12:06:35.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:35.598 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-26T12:06:35.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:35.625 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-26T12:06:35.634 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-26T12:06:35.647 INFO:teuthology.task.internal:Duration was 1325.364697 seconds 2024-05-26T12:06:35.647 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-26T12:06:35.662 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-26T12:06:35.662 DEBUG:teuthology.orchestra.run.smithi156:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-26T12:06:35.665 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-26T12:06:35.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-26T12:06:35.701 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T12:06:35.702 INFO:teuthology.orchestra.run.smithi156.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T12:06:35.703 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-26T12:06:36.082 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-26T12:06:36.082 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi156.front.sepia.ceph.com 2024-05-26T12:06:36.082 DEBUG:teuthology.orchestra.run.smithi156:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-26T12:06:36.143 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2024-05-26T12:06:36.143 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-26T12:06:36.173 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-05-26T12:06:36.174 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-26T12:06:36.202 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-26T12:06:36.203 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T12:06:36.205 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T12:06:36.216 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-26T12:06:36.267 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-26T12:06:36.267 DEBUG:teuthology.orchestra.run.smithi156:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-26T12:06:36.271 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-26T12:06:36.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-26T12:06:36.705 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-26T12:06:36.715 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-26T12:06:36.715 DEBUG:teuthology.orchestra.run.smithi156:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-26T12:06:36.740 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-26T12:06:36.764 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-26T12:06:36.789 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-26T12:06:36.798 DEBUG:teuthology.orchestra.run.smithi156:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.801 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.807 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.821 INFO:teuthology.orchestra.run.smithi156.stdout:kernel.core_pattern = core 2024-05-26T12:06:36.826 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2024-05-26T12:06:36.852 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-05-26T12:06:36.882 DEBUG:teuthology.orchestra.run.smithi156:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:36.925 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:36.939 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-26T12:06:36.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-26T12:06:36.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-26T12:06:36.963 INFO:teuthology.task.internal:Transferring archived files... 2024-05-26T12:06:36.964 DEBUG:teuthology.misc:Transferring archived files from smithi156:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi156 2024-05-26T12:06:36.964 DEBUG:teuthology.orchestra.run.smithi156:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-26T12:06:37.008 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi186 2024-05-26T12:06:37.008 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-26T12:06:37.041 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/leonidus-2024-05-26_10:47:39-fs-wip-lusov-quiesce-distro-default-smithi/7726315/remote/smithi192 2024-05-26T12:06:37.041 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-26T12:06:37.075 INFO:teuthology.task.internal:Removing archive directory... 2024-05-26T12:06:37.075 DEBUG:teuthology.orchestra.run.smithi156:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-26T12:06:37.077 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-26T12:06:37.079 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-26T12:06:37.124 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-26T12:06:37.136 INFO:teuthology.task.internal:Not uploading archives. 2024-05-26T12:06:37.136 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-26T12:06:37.168 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-26T12:06:37.168 DEBUG:teuthology.orchestra.run.smithi156:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-26T12:06:37.170 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-26T12:06:37.173 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-26T12:06:37.187 INFO:teuthology.orchestra.run.smithi186.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 26 12:06 /home/ubuntu/cephtest 2024-05-26T12:06:37.187 INFO:teuthology.orchestra.run.smithi156.stdout: 266037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 26 12:06 /home/ubuntu/cephtest 2024-05-26T12:06:37.188 INFO:teuthology.orchestra.run.smithi192.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 26 12:06 /home/ubuntu/cephtest 2024-05-26T12:06:37.189 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-26T12:06:37.201 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-26T12:06:37.235 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-quiesce/with-quiesce 6-workunit/kernel_untar_build}} duration: 1325.3646974563599 failure_reason: No module named 'tasks.fs' flavor: default owner: scheduled_leonidus@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9b4e81fbc8cd4cc98feda2bf55b0f2ba status: fail success: false 2024-05-26T12:06:37.235 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-26T12:06:37.308 INFO:teuthology.run:FAIL